Nadia2013 research

  • View
    693

  • Download
    0

Embed Size (px)

Transcript

  • Nadia Barbara Figueroa Fernandez

    3D Computer Vision and Applications in Robotics and Multimedia

    Reconstruct your world

    Reconstruct yourself

  • BACKGROUND

    3D COMPUTER VISION

    APPLICATIONS IN ROBOTICS Research Projects at TU Dortmund Masters Thesis at DLR

    APPLICATIONS IN MULTIMEDIA Research Projects at NYU Abu Dhabi

    DLRs rollin JusEn Humanoid

    AGENDA

  • EducaEon and Research PosiEons

    BACKGROUND

  • Fundamentals

    1

    General DeniEon

    2

    My DeniEon

    3

    What if a point cloud?

    Generate 3D representaBons of the world from the viewpoint of a sensor, generally in the form of 3D point clouds.

    Ability of powered devices to acquire a real Bme picture of the world in three dimensions. - Wikipedia

    3D COMPUTER VISION

    pP

    p = (x,y,z,r,g,b)A point cloud is a set of points where .

  • Primesense 3D sensor MicrosoP Kinect

    Example text

    3 Light Coding Structured Light

    Stereo Systems

    MulB-Camera Stereo

    2 TriangulaEon-based Systems 1 Time-Of-Flight Sensors

    Sensing Devices

    3D COMPUTER VISION

    LIDAR (Light DetecBon and Ranging) Radar Sonar

    TOF Cameras PMD (Photonic Mixing Device)

  • APPLICATIONS IN ROBOTICS

    CalibraEon and VericaEon Mapping and NavigaEon

    Object RecogniEon and Mobile ManipulaEon

  • Nadia Figueroa and JiVu Kurian

    OBJECT RECOGNITION FOR A MOBILE MANIPULATION PLATFORM

    GOAL: Detect and esBmate the pose of a wanted object in a table top scenario.

    PROPOSED APPROACH: Use CCD and PMD cameras. PRE-REQUISITES:

    1.- CalibraBon of PMD-CCD Camera Rig 2.- Object Database

  • Pre-Requisite 1: CalibraEon of PMD-CCD rig

    OBJECT RECOGNITION FOR A MOBILE MANIPULATION PLATFORM

    CalibraEon and camera set-up (CCD-PMD) Binocular camera setup of

    PMD and CCD Camera. Stereo System CalibraBon

    Method. MathemaBcally align the 2

    cameras in 1 viewing plane. Using epipolar geometry,

    calculate essenBal and fundamental matrices.

  • Pre-Requisite 2: Object Database

    OBJECT RECOGNITION FOR A MOBILE MANIPULATION PLATFORM

    Object model generaEon Each object is matched with 20 training images. The keypoints (SURF) that are repeatedly matched are selected as the best keypoints. APer training each object, we get 100 keypoints per object.

    Object 1 Object 2 Object 3

  • Object RecogniEon Algorithm

    OBJECT RECOGNITION FOR A MOBILE MANIPULATION PLATFORM

  • PMD Data FlaVening and Variance SegmentaEon Algorithm

    OBJECT RECOGNITION FOR A MOBILE MANIPULATION PLATFORM

    Original PMD

    Segmented PMD Fla^ened PMD

  • OBJECT RECOGNITION FOR A MOBILE MANIPULATION PLATFORM

  • DLRS ROLLIN JUSTIN

    Built of light-weight structures and joints with mechanical compliances and exibiliEes.

    (+) Compliant behavior of the arm (-) Low posiEong accuracy at the TCP (Tool-Center-Point) end pose.

    Designed to interact with humans and unknown environments.

    How is this low posiEon accuracy compensated in this lightweight design?

    Using the torque sensors. (+) An approximaBon of a joints deecBon is obtained by:

    :measured torque :sBness coecient of the gear (-) This approx. is insucient. It cannot measure the remaining mechanical exibiliBes.

    i = i + i Ki

    K

  • ROLLIN JUSTINS LOW POSITION ACCURACY

  • MASTER THESIS MOTIVATION

    Problem

    Goal

    Requirements

    Create a vericaBon rouBne to idenBfy the maximum bounds of the TCP posiBoning errors of humanoid JusBns upper kinemaBc chains.

    The feasibility of moBon planning is highly dependent on the posiBoning accuracy.

    1. Avoid using any external sensory system. 2. Avoid any human intervenBon

  • Supervisors: Florian Schmidt and Haider Ali

    3D REGISTRATION FOR VERIFICATION OF HUMANOID JUSTINS UPPER BODY KINEMATICS

    TCP = TwhThaTatcpTCP measured by forward kinematics:

    TCP = TwhThsTstcpTCP measured by stereo vision system:

    Tstcp

    Ths

    Tatcp

    Tha

    TCP

    Twh

    TCP End-Pose Error:

    Proposed Approach: Use the on-board stereo vision system to esBmate the TCP end-pose.

  • 3D REGISTRATION FOR VERIFICATION OF HUMANOID JUSTINS UPPER BODY KINEMATICS

    3D point clouds of the hand from the stereo cameras.

    EsBmate TCP by using registraBon between a point cloud of the hand and a model.

    RegistraEon method evaluaEon 1. Keypoint extracBon (SIFT) & point-to-point correspondence. 2. Local descriptor (FPFH/SHOT/CSHOT) matching using Ransac-based correspondence search.

    Model GeneraEon

    Data AcquisiEon

    Pose EsEmaEon Model generated from an extended metaview registraBon method from a selected subset of views generated by analyzing the distribuBon of max/min depth values.

  • Data AcquisiEon: Dense 3D point cloud generated from Stereo

    3D REGISTRATION FOR VERIFICATION OF HUMANOID JUSTINS UPPER BODY KINEMATICS

  • 3D REGISTRATION FOR VERIFICATION OF HUMANOID JUSTINS UPPER BODY KINEMATICS

    Point Cloud Processing Pass-through lter (remove background). StaBsBcal Outlier Removal (remove outliers) Voxel Grid Filter (downsample).

  • 3D RegistraEon Methods

    3D REGISTRATION FOR VERIFICATION OF HUMANOID JUSTINS UPPER BODY KINEMATICS

  • Model GeneraEon

    3D REGISTRATION FOR VERIFICATION OF HUMANOID JUSTINS UPPER BODY KINEMATICS

  • Model GeneraEon

    3D REGISTRATION FOR VERIFICATION OF HUMANOID JUSTINS UPPER BODY KINEMATICS

    Extended Metaview RegistraEon Method Consists of 3 steps: Global Thresholding Process: Reject the views that lie in unstable areas. Next Best View Ordering Algorithm: Find an order for incrementally registering the subset of point clouds. Metaview RegistraEon: The resulBng subset of views are registered and merged.

  • VericaEon RouEne

    3D REGISTRATION FOR VERIFICATION OF HUMANOID JUSTINS UPPER BODY KINEMATICS

    ek = et ,e

    fk = 3dRMS

    E = (e1,..,eN )

    F = ( f1,..., fN )

    F* = RANSAC(F)

    eb = max(et E*),max(e E*)

  • VericaEon RouEne

    3D REGISTRATION FOR VERIFICATION OF HUMANOID JUSTINS UPPER BODY KINEMATICS

  • Method EvaluaEon (Ground Truth)

    3D REGISTRATION FOR VERIFICATION OF HUMANOID JUSTINS UPPER BODY KINEMATICS

    Pose EsEmaEon using IR ART tracking system (Ground Truth)

    ART System Set-up MulB-camera setup that

    esBmates the 6DOF pose of the tracking targets.

    Mean accuracy of 0.04 pixels.

    Speed of 100 fps.

  • Method EvaluaEon (Ground Truth)

    3D REGISTRATION FOR VERIFICATION OF HUMANOID JUSTINS UPPER BODY KINEMATICS

    Implicit loop closure with tracking system (Ground Truth) By expressing in ART coordinate system a double loop closure is generated.

    TCPfk = TartheTTheTh ThaTatcp

    TCPreg = TartheTTheTh ThsTstcp

    TCPart = (TartheTTheTh )1TarthaTThaTtcp

    Error IdenBcaBon

    Tatcp

    Tha

    TCP

    ART

    TartheT

    TarthaT

    ThaTtcp

    TheTh

    Tstcp

    Ths

    TCPfk,TCPreg

  • Two step calibraEon: I. Center of RotaEon EsEmaEon: Non-rigid geometrically constrained sphere-mng

    min subject to :spherical t :measurements :spherical constraint II. Axis of RotaEons EsEmaEon Combined plane/circle mng for each axis.

    min

    :planar :radial

    3D REGISTRATION FOR VERIFICATION OF HUMANOID JUSTINS UPPER BODY KINEMATICS

    CalibraEon of Tracking targets to JusEn The esBmaBon of relies on the idenBcaBon of and

    TCPart

    TheTh

    ThaTtcp

    f = (k2 + k2)k=1

    N

    k =||vk m ||2 r2

    uTDTDu

    uTCu =1

    k

    k

    u

    C

    D

  • 3D REGISTRATION FOR VERIFICATION OF HUMANOID JUSTINS UPPER BODY KINEMATICS

    CalibraEon of Tracking targets to JusEn (contd) Create spherical trajectories around and .

    CoR is the posiBon of the joint deviaBons throughout 10 calibraBons. AoRs are the rotaBons

    Moun*ng frames: deviaBons throughout 10 calibraBons.

    R = [AoRx,AoRy,AoRz]

    t = [mx,my,mz ]T

    head

    TCP

    ThaTtcp = TCP(R,t)1TarthaT

    TheTh = head(R,t)1TartheT

    ThaTtcp

    TheTh

  • Method EvaluaEon (Ground Truth)

    3D REGISTRATION FOR VERIFICATION OF HUMANOID JUSTINS UPPER BODY KINEMATICS

  • Method EvaluaEon (Ground Truth)

    3D REGISTRATION FOR VERIFICATION OF HUMANOID JUSTINS UPPER BODY KINEMATICS

  • Experimental Results (TranslaEonal Error)

    3D REGISTRATION FOR VERIFICATION OF HUMANOID JUSTINS UPPER BODY KINEMATICS

  • Experimental Results (RotaEonal Error)

    3D REGISTRATION FOR VERIFICATION OF HUMANOID JUSTINS UPPER BODY KINEMATICS

  • Nadia Figueroa and Haider Ali (DLR)

    SEGMENTATION AND POSE ESTIMATION OF PLANAR METALLIC OBJECTS

    PROBLEM: Pose esBmaBon of planar metallic objects in a pile.

    PROPOSED APPROACH: (i) SegmentaBon using Euclidean clustering (ii) Pose EsBmaBon using RegistraBon

  • SEGMENTATION AND POSE ESTIMATION OF PLANAR METALLIC OBJECTS

    3D point clouds of the cloud from a range sensor.

    Cluster RegistraEon

    Eucli