Computervisionforrobotics

VictorEruhimovCTO,itseez

http://www.itseez.com

Whydoweneedcomputervision?

• Smartvideosurveillance• Biometrics• AutomaticDriverAssistanceSystems• Machinevision(Visualinspection)• Imageretrieval(e.g.GoogleGoggles)• Movieproduction• Robotics

Visionishard!Evenforhumans…

Texai parking

Agenda• Cameramodel• Stereovision

– StereovisiononGPU

• Objectdetectionmethods– Slidingwindow– Localdescriptors

• Applications– Texturedobjectdetection– Outletdetection– Visualodometry

Pinholecameramodel

€

x

€

y€

z

€

u€

v

€

xyz

"

#

$ $ $

%

&

' ' '

= RXYZ

"

#

$ $ $

%

&

' ' '

+ t

( x = x z( y = y z

u = fx ( x + cx

v = fy ( y + cy

Distortionmodel

Reprojection error

Homography

Perspective-n-Pointsproblem

• P4P• RANSAC(RANdom SAmple Consensus)

Stereo:epipolar geometry

( ) 01

1,, =⎟⎟⎟

⎠

⎞

⎜⎜⎜

⎝

⎛

⋅⋅ R

R

LL yx

Fyx

Fundamental matrix constraint

StereoRectification• Algorithmstepsareshownatright:• Goal:

– Eachrowoftheimagecontainsthesameworldpoints– “Epipolar constraint”

12

Result:Epipolaralignmentof features:

All:GaryBradskiandAdrianKaehler:LearningOpenCV

Stereocorrespondence

• Blockmatching• Dynamicprogramming• Inter-scanline dependencies

– Segmentation– Beliefpropagation

Stereocorrespondenceblockmatching

For each block in left image:

Search for the corresponding block in the right image such that SSD or SAD between pixel intensities is minimum

Pre- andpostprocessing

• Lowtexturefiltering• SSD/SADminimumambiguityremoval

• Usinggradientsinsteadofintensities

• Specklefiltering

Stereo Matching

Parallelimplementationofblockmatching

• Theoutercycleiteratesthroughdisparityvalues

• WecomputeSSDandcompareitwiththecurrentminimumforeachpixelinatile

• Differenttilesreusetheresultsofeachother

17

Parallelizationscheme

18

Optimizationconcepts

• Notusingtexture– savingregisters• 1threadper8pixelsprocessing– usingcache• Reducingtheamountofarithmeticoperations

• Non-parallelizablefunctions(specklefiltering)aredoneonCPU

19

Performancesummary

• CPU(i57502.66GHz),GPU(Fermicard448cores)

• BlockmatchingonCPU+2xGPUisthanCPUimplementation

withSSEoptimization,enablingreal-timeprocessingofHDimages!

Full-HDstereoinrealtime

http://www.youtube.com/watch?v=ThE7sRAtaWU

Applicationsofstereovision

• Machinevision• AutomaticDriverAssistance• Movieproduction• Robotics

– Objectrecognition– Visualodometry /SLAM

Objectdetection

Slidingwindowapproach

Cascadeclassifier

Stage1 Stage2 Stage3

image face face

Not face Not face Not face

face

Real-timeinyear2000!

Facedetection

Objectdetectionwithlocaldescriptors

• Detectkeypoints• Calculatelocaldescriptorsforeachpoint• Matchdescriptorsfordifferentimages• Validatematcheswithageometrymodel

FASTfeaturedetector

Keypoints example

SIFTdescriptor

David Lowe, 2004

SURFdescriptor

• 4x4squareregionsinsideasquarewindow20*s

• 4valuespersquareregion

Moredescriptors

• Onewaydescriptor• C-descriptor,FERNS,BRIEF• HoG• Daisy

Matchingdescriptorsexample

Waystoimprovematching

• Increasetheinlierstooutliersratio– Distancethreshold– Distanceratiothreshold(secondtofirstNNdistance)– Backward-forwardmatching– Windowedmatching

• Increasetheamountofinliers– Onetomanymatching

RandomSampleConsensus

• Don iterationsuntil#inliers>inlierThreshold– Drawk matchesrandomly– Findthetransformation– Calculateinlierscount– Rememberthebestsolution

Thenumberofiterationsrequired~

Geometryvalidation

Scalingup

• FLANN(FastLibraryforApproximateNearestNeighbors)– InOpenCV thankstoMariusMuja

• BagofWords– InOpenCV thankstoKenChatfield

• Vocabularytrees– IsgoingtobeinOpenCV thankstoPatrickMihelich

Projects

• Texturedobjectdetection• PR2robotautomaticplugin• Visualodometry /SLAM

Texturedobjectdetection

ObjectdetectionexampleIryna Gordon and David G. Lowe, "What and where: 3D object recognition with accurate pose," in Toward Category-Level Object Recognition, eds. J. Ponce, M. Hebert, C. Schmid, and A. Zisserman, (Springer-Verlag, 2006), pp. 67-82.

Manuel Martinez Torres, Alvaro ColletRomea, and Siddhartha Srinivasa, MOPED: A Scalable and Low Latency Object Recognition and Pose Estimation System, Proceedings of ICRA 2010, May, 2010.

Keypoint detection

• Wearelookingforsmalldarkregions

• Thisoperationtakesonly~10mson640x480image

• Therestofthealgorithmworksonlywithkeypoint regions

ItseezLtd.http://itseez.com

Classificationwithonewaydescriptor

• IntroducedbyHinterstoisser etal(TechnicalUofMunich,EcolePolytechnique)atCVPR2009

• Atestpatchiscomparedtosamplesofaffine-transformedtrainingpatcheswithEuclideandistance

• Theclosestpatchtogetherwithaposeguessarereconstructed

ItseezLtd.http://itseez.com

Keypoint classificationexamples• Onewaydescriptordoesthemostoftheoutletdetectionjobforus.Fewholesaremisclassified Ground hole

Power holeNon-hole keypointfrom outlet image

Background keypoint

ItseezLtd.http://itseez.com

Objectdetection

• Objectposeisreconstructedbygeometryvalidation(usinggeomertichashing)

ItseezLtd.http://itseez.com

Outletdetection:challengingcases

üShadows

üSevere lighting conditions

üPartial occlusions

ItseezLtd.http://itseez.com

PR2plugin (outletandplugdetection)

http://www.youtube.com/watch?v=GWcepdggXsU

Visualodometry

Visualodometry (II)

Morefun