OpenCVTutorial

VictorEruhimovItseez,CTO

http://opencv.willowgarage.comwww.willowgarage.com

www.itseez.com

Itseez

Outline

• OpenCVOverview• Cheatsheet• SimplePrograms• Tour• Features2D• Applications

2

OpenCV Czar

OpenCVOverview:OpenCVOverview:

General Image Processing Functions

Machine Learning:• Detection,• Recognition

Segmentation

Tracking

Matrix Math

Utilities and Data Structures

Fitting

Image Pyramids

Camera calibration,Stereo, 3D

TransformsFeatures

Geometric descriptors

Robotsupport

opencv.willowgarage.com >500algorithms

4GaryBradski

CLASSIFICATION/REGRESSION

CLUSTERING

TUNING/VALIDATION

CLASSIFICATION/REGRESSION(new)FastApproximateNN(FLANN)(new)ExtremelyRandomTreesCARTNaïveBayesMLP(Backpropagation) StatisticalBoosting,4flavorsRandomForestsSVMFaceDetector(Histogrammatching) (Correlation)

CLUSTERINGK-MeansEM(Mahalanobisdistance)

TUNING/VALIDATIONCrossvalidationBootstrappingVariableimportanceSamplingmethods

Machine Learning Library (MLL) AACBAABBCBCC

AAA

AACACB CBABBC

CCB

BCC

ABBCCB

BC A BBC

CBB

55http://opencv.willowgarage.com

OpenCV History

6GaryBradski

Willow

10

5

0

• Originalgoal:• Acceleratethefieldbyloweringthebartocomputervision• FindcompellingusesfortheincreasingMIPSoutinthemarket

• Timeline:

• Staffing:• Climbedin1999toaverage7firstcoupleofyears• Starting2003supportdeclinedbetweenzeroandonewithexception

oftransferringthemachinelearningfrommanufacturingworkIled(equivalentof3people).

• SupporttozerothecoupleofyearsbeforeWillow.• 5peopleoverthelastyear 6

NewDirectoryStructure• Re-Organizedintermsofprocessingpipelines• Codesite:https://code.ros.org/gf/project/opencv/– Core– Calibration,features,I/O,imgprocessing– MachineLearning,Obj.Rec– Python

• ~2.5Mdownloads

OtherLanguages

OpenCVConceptualStructure

Python

Java(TBD)

Machinelearning

HighGUI

SSETBBGPUMPU

Modules

CORECORE

imgprocimgproc

ObjectDetection Features2d Calib3d

Stereo

VOSLAM(TBD)

Stitching(TBD)

UserContrib

OperatingsystemOperatingsystem

C

C++

OpenCV Tends Towards Real Time

http://opencv.willowgarage.com

SoftwareEngineering• Workson:

– Linux,Windows,MacOS

• Languages:– C++,Python,C

• Onlinedocumentation:– Onlinereferencemanuals:C++,C andPython.

• We’vebeenexpandingUnittestcode• WillsoonstandardizeoncxxorGoogle’stestsystem.• TESTCOVERAGE:

License• BasedonBSDlicense• Freeforcommercialorresearchuse

– Inwholeorinpart– Doesnotforceyourcodetobeopen– Youneednotcontributeback

• Wehopeyouwillcontributeback,recentcontribution,C++wrapperclassusedforGoogleStreetMaps*

GaryBradski(c)2008 11

*ThankstoDanielFilip11GaryBradski,2009

What’saddedinDecember2010OpenCV 2.2?

• Detector/Descriptorpipeline(Features2D)– Manysupportingdetectorsanddescriptorfeatures

• EasyinterfacetoPascalVOC• BOWandLatentSVMclassificationengines• ExperimentalUserContrib• Focusdetector?• Visualization(“HighGUI”)willbebasedonQt• OfficialsupportofAndroidOS• UpdatedFLANNlibrary• LimitedCudasupport(stereo)

What’sinProgress?

• GPUsupportthroughoutthelibrary• Morefunctionalityinfeatures2d• Betterposeestimationalgorithms

– ePnP– Stereoposeestimation

• Circlespatterndetector• BettersupportofAndroid• Supportforgoogle test

WhereisOpenCVUsed?

2Mdownloads

•Wellover2Mdownloads

ScreenshotsbyGaryBradski,2005

• GoogleMaps,Googlestreetview,GoogleEarth,Books• AcademicandIndustryResearch• Safetymonitoring(Damsites,mines,swimmingpools)

• Securitysystems• Imageretrieval• Videosearch• Structurefrommotioninmovies• Machinevisionfactoryproductioninspectionsystems• Robotics

Useful OpenCV Links

1515GaryBradski,2009

OpenCVWiki:http://opencv.willowgarage.com/wiki

OpenCVCodeRepository:svn co https://code.ros.org/svn/opencv/trunk/opencv

NewBookonOpenCV:http://oreilly.com/catalog/9780596516130/

Or,directfromAmazon:http://www.amazon.com/Learning-OpenCV-Computer-Vision-Library/dp/0596516134

Codeexamplesfromthebook:http://examples.oreilly.com/9780596516130/

Documentationhttp://opencv.willowgarage.com/documentation/index.html

UserGroup(39717members):http://tech.groups.yahoo.com/group/OpenCV/join

Outline

• OpenCVOverview• Cheatsheet• SimplePrograms• Tour• Features2D• Applications

GaryBradski,2009 16

MainStructures

New“Image”:cv::Mat

Matdoesreferencecounting, soitdoestherightthingwhenitgoesoutofscopeyoucanalsoeasilymakestlvectortsormapsoutofMat.

MatareSimple

MatM(480,640,CV_8UC3);//Makea640x480imgRectroi(100,200,20,40);//MakearegionofintMatsubM=M(roi);//Takeasubregion,

//nocopyisdone

Mat_<Vec3b>::iteratorit=subM.begin<Vec3b>(),itEnd=subM.end<Vec3b>();

//0outplacesinsubMwhereblue>redfor(;it!=itEnd;++it)if((*it)[0]>(*it)[2])(*it)[0]=0;

MatrixManipulation

SimpleMatrixOperations

SimpleImageProcessing

ImageConversions

Histogram

I/O

SerializationI/O

SerializationI/O

GUI(“HighGUI”)

CameraCalibration,Pose,Stereo

ObjectRecognition

samples/c

31

In...\opencv\samples\c

bgfg_codebook.cpp - Useofaimagevaluecodebookforbackgrounddetectionforcollectingobjects

bgfg_segm.cpp - Useofabackgroundlearningengineblobtrack.cpp - Engineforblobtrackinginimagescalibration.cpp - CameraCalibrationcamshiftdemo.c - Useofmeanshiftinsimplecolortrackingcontours.c - Demonstrateshowtocomputeanduse

objectcontoursconvert_cascade.c - Changethewindowsizeinarecognition

cascadeconvexhull.c - Findtheconvexhullofanobjectdelaunay.c - Triangulatea2Dpointclouddemhist.c - Showhowtousehistogramsforrecognitiondft.c - Discretefouriertransformdistrans.c - distancemapfromedges inanimagedrawing.c - Variousdrawingfunctionsedge.c - Edgedetectionfacedetect.c - Facedetectionbyclassifiercascadeffilldemo.c - Floodfillingdemofind_obj.cpp - DemouseofSURFfeaturesfitellipse.c - Robustelipse fittinghoughlines.c - Linedetectionimage.cpp - Showsuseofnewimageclass,CvImage();inpaint.cpp - Textureinfilltorepairimagerykalman.c - Kalman filterfortrackignkmeans.c - K-Meanslaplace.c - Convolveimagewithlaplacian.

letter_recog.cpp - ExampleofusingmachinelearningBoosting,Backpropagation(MLP)andRandomforests

lkdemo.c - Lukas-Canadaopticalflowminarea.c - Foracloudofpointsin2D,findmin

boundingboxandcircle.ShowsuseofCv_SEQ

morphology.c - DemonstratesErode,Dilate,Open,Closemotempl.c - Demonstratesmotiontemplates

(orthogonalopticalflowgivensilhouettes)mushroom.cpp - Demonstratesuseofdecisiontrees(CART)

forrecognitionpyramid_segmentation.c - Colorsegmentationinpyramidsquares.c - Usescontourprocessingtofindsquares

inan imagestereo_calib.cpp - Stereocalibration,recognitionanddisparity

mapcomputationwatershed.cpp - Watershedtransformdemo.

samples/C++

32

Samples/python

BookExamples

GaryBradski,2009 34

ch2_ex2_1.cpp Loadimagefromdiskch2_ex2_2.cpp Playvideofromdiskch2_ex2_3.cpp Addaslidercontrolch2_ex2_4.cpp Load,smoothanddsiplayimagech2_ex2_5.cpp Pyramiddownsamplingch2_ex2_6.cpp CvCannyedgedetectionch2_ex2_7.cpp PyramiddownandCannyedgech2_ex2_8.cpp Aboveprogramsimplifiedch2_ex2_9.cpp Playvideofromcameraorfilech2_ex2_10.cpp Readandwritevideo,doLogpolar

ch3_ex3_1.txt Matrixstructurech3_ex3_2.txt Matrixcreationandreleasech3_ex3_3.cpp Creatematrixfromdatalistch3_ex3_4.cpp AccessingmatrixdataCV_MAT_ELEM()ch3_ex3_5.cpp SettingmatrixCV_MAT_ELEM_PTR()ch3_ex3_6.txt Pointeraccesstomatrixdata

ch3_ex3_7.txt ImageandMatrixElementaccessfunctionsch3_ex3_8.txt Settingmatrixorimageelementsch3_ex3_9.cpp Summingallelementsin3channelmatrixch3_ex3_10.txt IplImageHeaderch3_ex3_11.cpp Useofwidthstepch3_ex3_12.cpp UseofimageROIch3_ex3_13.cpp ImplementinganROIusingwidthstepch3_ex3_14.cpp Alphablendingexamplech3_ex3_15.cpp SavingandloadingaCvMatch3_ex3_16.txt Filestoragedemoch3_ex3_17.cpp WritingconfigurationfilesasXMLch3_ex3_19.cpp ReadinganXMLfilech3_ex3_20.cpp HowtocheckifIPPaccelerationison

BookExamples

GaryBradski,2009 35

ch4_ex4_1.cpp Useamousetodrawboxesch4_ex4_2.cpp Useatrackbarasabuttonch4_ex4_3.cpp Findingthevideocodec

ch5_ex5_1.cpp UsingCvSeqch5_ex5_2.cpp cvThresholdexamplech5_ex5_3.cpp Combiningimageplanesch5_ex5_4.cpp Adaptivethreshiolding

ch6_ex6_1.cpp cvHoughCirclesexamplech6_ex6_2.cpp Affinetransformch6_ex6_3.cpp Perspectivetransformch6_ex6_4.cpp Log-Polarconversionch6_ex6_5.cpp 2DFourierTransform

ch7_ex7_1.cpp Usinghistogramsch7_ex7_2.txt EarthMover’sDistanceinterfacech7_ex7_3_expanded.cpp EarthMover’sDistancesetupch7_ex7_4.txt UsingEarthMover’sDistancech7_ex7_5.cpp Templatematching/CrossCorr.ch7_ex7_5_HistBackProj.cpp Backprojectionofhistograms

ch8_ex8_1.txt CvSeqstructurech8_ex2.cpp Contourstructurech8_ex8_2.cpp Findingcontoursch8_ex8_3.cpp Drawingcontours

BookExamples

GaryBradski,2009 36

ch9_ex9_1.cpp Samplingfromalineinanimagech9_watershed.cpp ImagesegmentationusingWatershedtransformch9_AvgBackground.cpp Backgroundmodelusinganaverage imagech9_backgroundAVG.cpp Backgroundaveragingusingacodebookcomparedtojustanaveragech9_backgroundDiff.cpp Usethecodebookmethodfordoingbackgrounddifferencingch9_ClearStaleCB_Entries.cpp Refinecodebooktoeliminatestaleentriescv_yuv_codebook.cpp CorecodeusedtodesignOpenCVcodebook

ch10_ex10_1.cpp OpticalflowusingLucas-Kanadeinanimagepyramidch10_ex10_1b_Horn_Schunck.cpp OpticalflowbasedonHorn-Schunckblockmatchingch10_ex10_2.cpp Kalmanfilterexamplecodech10_motempl.cpp Usingmotiontemplatesforsegmentingmotion.

ch11_ex11_1.cpp Cameracalibrationusingautomaticchessboardfindingusingacamerach11_ex11_1_fromdisk.cpp Doingthesame,butreadfromdiskch11_chessboards.txt Listofincludedchessboardsforcalibrationfromdiskexample

ch12_ex12_1.cpp Creatingabird’seyeviewofascene usinghomographych12_ex12_2.cpp ComputingtheFundamentalmatrixusingRANSACch12_ex12_3.cpp Stereocalibration,rectificationandcorrespondencech12_ex12_4.cpp 2Drobustlinefittingch12_list.txt ListofincludedstereoL+Rimagepairdata

ch13_dtree.cpp Exampleofusingadecisiontreech13_ex13_1.cpp Usingk-meansch13_ex13_2.cpp Creatingandtrainingadecisiontreech13_ex13_3.cpp Trainingusingstatisticalboostingch13_ex13_4.cpp FacedetectionusingViola-Jonescvx_defs.cpp Somedefinesforusewithcodebooksegmentatio

PythonFaceDetectorNode:1

37

#!/usr/bin/python"""Thisprogramisdemonstrationpython ROSNodeforfaceandobjectdetectionusinghaar-likefeatures.Theprogramfindsfacesinacameraimageorvideostreamanddisplaysaredboxaround them.Python implementationby:RomanStanchak,JamesBowman"""importroslibroslib.load_manifest('opencv_tests')importsysimportosfromoptparseimportOptionParserimportrospyimportsensor_msgs.msgfromcv_bridgeimportCvBridgeimportcv

#Parametersforhaardetection#FromtheAPI:#Thedefaultparameters(scale_factor=2,min_neighbors=3,flags=0)aretuned#foraccurateyetslowobjectdetection.Forafasteroperationonrealvideo#imagesthesettingsare:#scale_factor=1.2,min_neighbors=2,flags=CV_HAAR_DO_CANNY_PRUNING,#min_size=<minimumpossiblefacesize

min_size=(20,20)image_scale=2haar_scale=1.2min_neighbors=2haar_flags=0

TheSetup

PythonFaceDetectorNode:2

38

if__name__ == '__main__':

pkgdir=roslib.packages.get_pkg_dir("opencv2")haarfile=os.path.join(pkgdir,"opencv/share/opencv/haarcascades/haarcascade_frontalface_alt.xml")

parser=OptionParser(usage="usage:%prog[options][filename|camera_index]")parser.add_option("-c","--cascade", action="store",dest="cascade", type="str",help="Haarcascadefile,default%default", default=haarfile)(options,args)=parser.parse_args()

cascade=cv.Load(options.cascade)br=CvBridge()

defdetect_and_draw(imgmsg):img=br.imgmsg_to_cv(imgmsg,"bgr8")#allocatetemporaryimagesgray=cv.CreateImage((img.width,img.height),8,1)small_img=cv.CreateImage((cv.Round(img.width/image_scale),

cv.Round(img.height/image_scale)),8,1)

#convertcolorinputimagetograyscalecv.CvtColor(img,gray,cv.CV_BGR2GRAY)

#scaleinputimageforfasterprocessingcv.Resize(gray,small_img,cv.CV_INTER_LINEAR)

cv.EqualizeHist(small_img,small_img)

if(cascade):faces=cv.HaarDetectObjects(small_img,cascade,cv.CreateMemStorage(0),

haar_scale,min_neighbors,haar_flags,min_size)iffaces:for((x,y,w,h),n)infaces:#theinputtocv.HaarDetectObjectswasresized,soscalethe#boundingboxofeachfaceandconvertittotwoCvPointspt1=(int(x*image_scale),int(y*image_scale))pt2=(int((x+w)*image_scale),int((y+h)*image_scale))cv.Rectangle(img,pt1,pt2,cv.RGB(255,0,0),3,8,0)

cv.ShowImage("result",img)cv.WaitKey(6)

rospy.init_node('rosfacedetect')image_topic=rospy.resolve_name("image")rospy.Subscriber(image_topic,sensor_msgs.msg.Image,detect_and_draw)rospy.spin()

TheCore

Outline

• OpenCV Overview• Cheatsheet• SimplePrograms• Tour• Features2D• Applications

GaryBradski,2009 39

NewC++API:UsageExample

doublecalcGradients(constIplImage*src,intaperture_size=7){CvSizesz=cvGetSize(src);IplImage*img16_x=cvCreateImage(sz,IPL_DEPTH_16S,1);IplImage*img16_y=cvCreateImage(sz,IPL_DEPTH_16S,1);

cvSobel(src,img16_x,1,0,aperture_size);cvSobel(src,img16_y,0,1,aperture_size);

IplImage*imgF_x=cvCreateImage(sz,IPL_DEPTH_32F,1);IplImage*imgF_y=cvCreateImage(sz,IPL_DEPTH_32F,1);

cvScale(img16_x,imgF_x);cvScale(img16_y,imgF_y);

IplImage*magnitude=cvCreateImage(sz,IPL_DEPTH_32F,1);cvCartToPolar(imgF_x,imgF_y,magnitude);double res=cvSum(magnitude).val[0];

cvReleaseImage(&magnitude);cvReleaseImage(&imgF_x);cvReleaseImage(&imgF_y);cvReleaseImage(&img16_x);cvReleaseImage(&img16_y);

returnres;}

doublecontrast_measure(constMat&img){Matdx,dy;

Sobel(img,dx,1,0,3,CV_32F);Sobel(img,dy,0,1,3,CV_32F);magnitude(dx,dy,dx);

returnsum(dx)[0];}

FocusDetector

C: C++:

40

Pyramid/**Makeanimagepyramidwithlevelsofarbitraryscalereduction (0,1)*M Input image*reduction Scalingfactor1>reduction>0*levels Howmanylevelsofpyramid*pyr stdvectorcontainingthepyramid*sz Thewidthandheightofblurringkernel,DEFAULT3*sigma ThestandarddeviationoftheblurringGaussianDEFAULT0.5*RETURNS Numberoflevelsachieved*/intbuildGaussianPyramid(constMat&M,double reduction,intlevels,

vector<Mat>&pyr, intsz=3,floatsigma=0.5){if(M.empty())return0;pyr.clear();//Clearitupif((reduction<=0.0)||(reduction>=1.0))return0;MatMblur,Mdown=M;pyr.push_back(Mdown);Sizeksize=Size(sz,sz);intL=1;for(;L<=levels;++L){if((reduction*Mdown.rows)<=1.0||(reduction*Mdown.cols)<=1.0)break;GaussianBlur(Mdown,Mblur,ksize,sigma,sigma);resize(Mblur,Mdown,Size(),reduction,reduction);pyr.push_back(Mdown);

}returnL;

}

Outline

• OpenCV Overview• Cheatsheet• SimplePrograms• Tour• Features2D• Applications

GaryBradski,2009 42

Canny Edge DetectorCanny Edge Detector

43

DistanceTransformDistanceTransform• Distancefieldfromedgesofobjects

FloodFilling

44

Hough Transform

45GaryBradski,AdrianKahler2008

SpaceVariantvision:Log-PolarTransform

46

ScreenshotsbyGaryBradski,2005

Scale Space

voidcvPyrDown(IplImage* src,IplImage* dst,IplFilter filter=IPL_GAUSSIAN_5x5);

voidcvPyrUp(IplImage* src,IplImage* dst,IplFilter filter=IPL_GAUSSIAN_5x5);

47

ChartbyGaryBradski,2005

Thresholds

48

ScreenshotsbyGaryBradski,2005

Histogram Equalization

49ScreenshotsbyGaryBradski,2005

Contours

50

MorphologicalOperationsExamplesMorphologicalOperationsExamples

• Morphology- applyingMin-Max. Filtersanditscombinations

Opening IoB= (IΘB)⊕BDilatation I⊕BErosion IΘBImage I

Closing I•B= (I⊕B)ΘB TopHat(I)= I - (IΘB) BlackHat(I)= (I⊕B) - IGrad(I)= (I⊕B)-(IΘB)

Imagetextures

• Inpainting:• Removesdamagetoimages,inthiscase,itremovesthetext.

Segmentation

• Pyramid,mean-shift,graph-cut• Here:Watershed

5353

ScreenshotsbyGaryBradski,2005

• GraphCutbasedsegmentation

54

RecentAlgorithms:RecentAlgorithms: GrabCut

ImagesbyGaryBradski,©2010

Motion Templates Motion Templates (work with James Davies)

• Objectsilhouette• Motionhistoryimages• Motionhistorygradients• Motionsegmentationalgorithm

silhouette MHI MHG

55

ChartsbyGaryBradski,2005

Segmentation,MotionTrackingand

GestureRecognition

PoseRecognition

MotionSegmentation

GestureRecognition

MotionSegmentation

ScreenshotsbyGaryBradski,2005

NewOpticalFlowAlgorithms

…)

// opencv/samples/c/lkdemo.cint main(…){…CvCapture* capture = <…> ?

cvCaptureFromCAM(camera_id) : cvCaptureFromFile(path);

if( !capture ) return -1;for(;;) {

IplImage* frame=cvQueryFrame(capture);if(!frame) break;// … copy and process image

cvCalcOpticalFlowPyrLK( …)cvShowImage( “LkDemo”, result );c=cvWaitKey(30); // run at ~20-30fps speedif(c >= 0) {

// process key}}cvReleaseCapture(&capture);}

lkdemo.c,190lines(needs cameratorun)

∑∑ ⎥⎦

⎤⎢⎣

⎡=

⎥⎥

⎦

⎤

⎢⎢

⎣

⎡=∂∂=∂

=∂⋅

⋅∂∂+⋅∂∂=∂∂−

=+++

y

xt

yyx

yxx

II

IbIII

IIIGyxX

bXG

dtdyyIdtdxxItItyxIdttdyydxxI

,,

,),,(

,

);/(/)/(//);,,(),,(

2

2

TrackingwithCAMSHIFT

• Controlgamewithhead

ScreenshotsbyGaryBradski,2005

Projections

ScreenshotsbyGaryBradski,2005

Stereo…DepthfromTriangulation

• Involvedtopic,herewewilljustskimthebasicgeometry.

• Imaginetwoperfectlyalignedimageplanes:

60

Depth“Z”anddisparity“d”areinverslyrelated:

Stereo• Inalignedstereo,depthisfromsimilartriangles:

• Problem:Camerasarealmostimpossibletoalign• Solution:Mathematicallyalignthem:

61

rl

rl

xxfTZ

ZT

fZxxT

−=⇒=

−

−− )(

All:GaryBradskiandAdrianKaehler:LearningOpenCV

StereoRectification• Algorithmstepsareshownatright:• Goal:

– Eachrowoftheimagecontainsthesameworldpoints– “Epipolar constraint”

62

Result:Epipolaralignmentof features:

All:GaryBradskiandAdrianKaehler:LearningOpenCV

Outline

• OpenCV Overview• Cheatsheet• SimplePrograms• Tour• Features2D• Applications

GaryBradski,2009 63

Features2dcontents

Detection

Detectorsavailable• SIFT• SURF• FAST• STAR• MSER• HARRIS• GFTT (Good Features

ToTrack)

Description

Descriptorsavailable• SIFT• SURF• Calonder• Ferns• Oneway

Matching

Matchersavailable• BruteForce• FlannBased• BOW

Matchesfilters(underconstruction)• Crosscheck• Ratiocheck

Detectorinterfaces

classFeatureDetector{public:virtual~FeatureDetector() {}

//Detectkeypoints inanimage.virtualvoiddetect(constMat&image,vector<KeyPoint>&keypoints,

constMat&mask=Mat())const=0;//Detectkeypoints inanimageset.voiddetect(constvector<Mat>&imageCollection,

vector<vector<KeyPoint>>&pointCollection,constvector<Mat>&masks=vector<Mat>())const;

virtualvoid read(constFileNode&fn){}virtualvoidwrite(FileStorage& fs )const{}

protected:…};

Creatingadetector

• StaticallySurfFeatureDetector detector;

• Usingclassfactorycv::Ptr<FeatureDetector>detector=createFeatureDetector(“SURF”);

Runningdetector

Matimg =imread("test.png");vector<KeyPoint>keypoints;

SurfFeatureDetectordetector;detector.detect(img,keypoints );

Descriptorinterfaces

• Fordescriptorsthatcanberepresentedasvectorsinmultidimensionalspace:DescriptorExtractor andDescriptorMatcher

• Moregeneralinterface(oneway,decision-tree-baseddescriptors):GenericDescriptorMatcher

DescriptorExtractor interfaces

classCV_EXPORTSDescriptorExtractor{public:virtual~DescriptorExtractor(){}//Computethedescriptorsforasetofkeypoints inanimage.virtualvoidcompute(constMat&image,vector<KeyPoint>&keypoints,

Mat&descriptors)const=0;//Computethedescriptorsforakeypoints collectiondetectedinimagecollection.voidcompute(constvector<Mat>&imageCollection,

vector<vector<KeyPoint>>&pointCollection,vector<Mat>&descCollection )const;

virtualvoidread(constFileNode&){}virtualvoidwrite(FileStorage&)const{}virtualint descriptorSize()const=0;virtualint descriptorType()const=0;

protected:…};

DescriptorExtractor creating

• StaticallySurfDescriptorExtractor descriptorExtractor;

• Usingclassfactorycv::Ptr<DescriptorExtractor>descriptorExtractor =createDescriptorExtractor("SURF");

DescriptorExtractor running

Ptr<FeatureDetector>detector=createFeatureDetector("FAST");

Ptr<DescriptorExtractor>descriptorExtractor =createDescriptorExtractor("SURF");

vector<KeyPoint>keypoints;detector->detect(img,keypoints );Matdescriptors;descriptorExtractor->compute(img,keypoints,descriptors);

DescriptorMatcher interfaces

• Twogroupsofmatchmethods– tomatchdescriptorsofimagepair– tomatchdescriptorsofoneimagetoimageset

• Eachgroupconsistsfromtreetypemethods– match()– knnMatch()– radiusMatch()

Matchingofimagepair//detectingkeypointsSurfFeatureDetector detector;vector<KeyPoint>keypoints1, keypoints2;detector.detect(img1,keypoints1 );detector.detect(img2,keypoints2 );

//computing descriptorsSurfDescriptorExtractor extractor;Matdescriptors1,descriptors2;extractor.compute(img1,keypoints1, descriptors1 );extractor.compute(img2,keypoints2, descriptors2 );

//matchingdescriptorsBruteForceMatcher<L2<float>>matcher;vector<DMatch>matches;matcher.match(descriptors1,descriptors2,matches);

Visualizekeypoints

Visualizematches

Matimg_points;drawKeypoints(img,keypoints,img_points );namedWindow(“keypoints”,1);imshow(“keypoints”,img_points );waitKey();

Matimg_matches;drawMatches(img1,keypoints1,

img2,keypoints2,img_matches);namedWindow(“matches”,1);imshow(“matches”,img_matches );waitKey();

Runningthesample

• DownloadOpenCV• Compile• Runmatcher_simple:

bin/matcher_simple ../../opencv/samples/c/box.png../../opencv/samples/c/box_in_scene.png

• Selectadetectorthatgivesthemaximumnumberofkeypoints

• SwitchSIFTandSURFdescriptors

Cross-checkoutliermatchfiltering

BruteForceMatcher<L2<float>>descriptorMatcher;vector<DMatch>filteredMatches12,matches12,matches21;descriptorMatcher.match(descriptors1,descriptors2,matches12);descriptorMatcher.match(descriptors2,descriptors1,matches21);

for(size_t i =0;i <matches12.size(); i++){DMatch forward=matches12[i];DMatch backward=matches21[forward.trainIdx];if(backward.trainIdx ==forward.queryIdx )filteredMatches12.push_back(forward);

}

Ratiotesttofiltermatches

]1,0(21∈=

MinDistMinDistRatio (lessisbetter)

outlierelseinlierthresholdRatioif ,)3.0( ⇒<

Calculatinginliers(planarobjectscase)

• Detectkeypoints• Findmatchesusingdescriptors• Filtermatchesusingcross-check• Calculatebesthomography• Filteroutliers• Run

bin/descriptor_extractor_matcher SURFSURF../../opencv/samples/c/box.png../../opencv/samples/c/box_in_scene.png3

Thelastparameteristhereprojection thresholdforRANSAC

Detectortestbench

• Measuresofdetectorrepeatabilityaretakenfrom– K.Mikolajczyk,Cordelia Schmid,“Scale&AffineInvariantInterestPoint

Detectors”,IJCV60(1),63–86,2004.– K.Mikolajczyk etal,AComparisonofAffineRegionDetectors,IJCV

65(1/2):43-72,2005.

• Testimagesaretakenfromhttp://www.robots.ox.ac.uk/~vgg/data/data-aff.html

• Testbench islocatedinopencv_extra/testdata/cv/detectors_descriptors_evaluation/detectors

Descriptortestbench

• Measuresofdescriptormatchingaccuracyaretakenfromhttp://www.robots.ox.ac.uk/~vgg/research/affine/det_eval_files/mikolaj

czyk_pami2004.pdf

• Testimagesaretakenfromhttp://www.robots.ox.ac.uk/~vgg/data/data-aff.html

• Testbench islocatedinopencv_extra/testdata/cv/detectors_descriptors_evaluation/descriptors

OpenCV andROS

• Opencv2packagetofetchandcompileopencv• Messages:

– sensor_msgs::Image– sensor_msgs::CameraInfo

• cv_bridge toconvertbetweenmessagesandimages

• image_geometry::PinholeCameraModel andimage_geometry::StereoCameraModel tomanage2d<->3dconversions

Q&A

• Foilswillbeavailableathttp://itseez.com