Upload
sumisumith
View
533
Download
3
Embed Size (px)
Citation preview
PowerPoint Presentation
CVPR 2016 Tutorial Fitting Surface Models to Data2016/10/02@sumisumith
1
2
http://awf.fitzgibbon.ie/cvpr16_tutorial
2
Fitting Surface Models to Data3CVPR 2016 Tutorial
Andrew Fitzgibbon, MicrosoftJonathan Taylor, PerceptiveIO
3
PeopleFinding Nemo: Deformable Object Class Modelling using Curve MatchingCVPR 10Mukta Prasad, Andrew Fitzgibbon, Andrew Zisserman, Luc Van GoolKintre: Animating the World with the Human BodyUIST 12Jiawen (Kevin) Chen, Shahram Izadi, FitzgibbonThe Vitruvian Manifold: Inferring dense correspondences for one-shot human pose estimationCVPR 12Jonathan Taylor, Jamie Shotton, Toby Sharp, FitzgibbonWhat shape are dolphins? Building 3D morphable models from 2D imagesPAMI 13Tom Cashman, FitzgibbonUser-Specific Hand Modeling from Monocular Depth SequencesCVPR 14Taylor, Richard Stebbing, Varun Ramakrishna, Cem Keskin, Shotton, Izadi, Fitzgibbon, Aaron Hertzmann Real-Time Non-Rigid Reconstruction Using an RGB-D CameraSIGGRAPH 14Michael Zollhfer, Matthias Niener, Izadi, Christoph Rhemann, Christopher Zach, Matthew Fisher, Chenglei Wu, Fitzgibbon, Charles Loop, Christian Theobalt, Marc StammingerLearning an Efficient Model of Hand Shape Variation from Depth ImagesCVPR 15Sameh Khamis, Taylor, Shotton, Keskin, Izadi, Fitzgibbon Efficient and Precise Interactive Hand Tracking through Joint, Continuous Optimization of PoseSIGGRAPH 16and CorrespondencesTaylor, Lucas Bordeaux, Cashman, Bob Corish, Keskin, Sharp, Eduardo Soto, David Sweeney, Julien Valentin, Ben Luff, Arran Topalian, Erroll Wood, Khamis, Kohli, Izadi, Richard Banks, Fitzgibbon, Shotton.Fits Like a Glove: Rapid and Reliable Hand Shape Personalization. CVPR 16David Joseph Tan, Cashman, Taylor, Fitzgibbon, Daniel Tarlow, Khamis, Izadi, Shotton.
4
Learn how to SOLVE HARD VISION PROBLEMS, using tools that may appear inelegant, but are much smarter than they look.Goal5
5
ApplicationsCurve/surface fitting
Parameter estimationBundle adjustment(Video from our friends at Google)
Kintre7
KinEtre: Animating the World with the Human Body Chen et al. UIST 2012
Kintre8
KinEtre: Animating the World with the Human Body Chen et al. UIST 2012
Kintre9
KinEtre: Animating the World with the Human Body Chen et al. UIST 2012
Vitruvian Manifold, CVPR 12
Fitting subdivision surfaces to 2D data
Fitting subdivision surfaces to 2D data
15
Fitting polygon meshes to video
17
[3D Scanning Deformable Objects with a Single RGBD Sensor, Dou et al, CVPR15]Input Kinect StreamKinectFusionDeformable Fusion
17
Hand Tracking
Hand Shape Personalization:CVPR 2014, CVPR 2015, CVPR 2016Discriminative Hand Pose ReinitializationICCV 2015, CHI 2015
Hand Pose Estimation via Model Fitting (read Hand Tracking)CHI 2015, SIGGRAPH 2016
Image Denoising19[STRANDMARK & AGARWAL, 2014, arXiv:1403.5590]
Matrix Factorization [Hong & F., ICCV 15]
Matrix Factorization [Hong & F., ICCV 15]
Myth: You dont need to optimize far22
For each task, the method is the same23
So, you can do lots of things by fitting models to data.
How do you do it right?
Lets look at some examples.
24
Continuous Optimization
Andrew FitzgibbonMicrosoft Research Cambridge
Goal
Classes of functionsquadraticconvexquasiconvexmulti-extremumnoisyhorrible
27
Classes of functionsquadraticconvexquasiconvexmulti-extremumnoisyhorrible
28
29quadraticconvexquasiconvexmulti-extremum
29
30quadraticconvexquasiconvexmulti-extremumEasyHard
30
Derivatives31
01332 440 955 o2 sian31
Alternation
EasyHard
33
Rosenbrock
33
Gradient DescentAlternation is slow because valleys may not be axis alignedSo try gradient descent?
-5
0
5
10
15
-5
0
5
10
15
Gradient DescentAlternation is slow because valleys may not be axis alignedSo try gradient descent?
Gradient DescentAlternation is slow because valleys may not be axis alignedSo try gradient descent?
Note that convergence proofs are available for both of the aboveBut so what?
And on a hard problem
Use a better algorithm(Nonlinear) conjugate gradientsUses 1st derivatives onlyAvoids undoing previous work
Use a better algorithm(Nonlinear) conjugate gradientsUses 1st derivatives onlyAnd avoids undoing previous work101 iterations on this problem
but we can do better
Use second derivatives
Use second derivatives
Use second derivativesHow does it look?
43
Use second derivatives
Use second derivatives
Use second derivatives
Use second derivatives
1st derivatives again
Back to first derivatives
50
Back to first derivatives
Order N Cubed?
Typical Hessian structure
53
54 http://bit.ly/2dcaFr7
Conclusion: YMMV
Conclusion: YMMVGIRAFFE
500 runs
Conclusion: YMMV
FACE
1000 runs
DINOSAUR
1000 runsGIRAFFE
500 runs
On many problems, alternation is just fineIndeed always start with a couple of alternation stepsComputing 2nd derivatives is a painBut you dont need to for LM
There is no universal optimizer
What is a surface?
59
00.20.40.60.81
1.922.12.22.32.4
Curvesfunction y(x::Interval)::Real = .3*x + 2
function C(t::Interval)::Point2D = Point2D(t^2 + 2, t^2 t + 1)
function S(u::Interval, v::Real)::Point3D = Point3D(cos(u), sin(u), v)
60
22.22.42.62.83
0.750.80.850.90.951
Surface61
61
Surface62
62
Surface63
63
TOOL: Subdivision Surfaces 64
And I still havent told you about the surface model
64
Control mesh65
Limit Surface66
Subdiv rule: Step 1. Add new vertices67
Subdiv rule: Step 2. Average neighbours68
2 subdivisions69
3 subdivisions70
Limit Surface71
Control vertices define the shape72
Subdivision Surface: Parametric form73
73
examples74
Back to Dolphins
75
What Shape are Dolphins? Building 3D Morphable Models from 2D Images
76
77Loop Subdivision Surface 3DBB3D
Non-rigid 3D B
3D
( Non-rigid )
77
Model representation
78
79
80
81
Data terms
82
Data terms83Camera positionProjectione.g. Perspective
Data terms84
Gaussian shape weights
Smooth contour
85
86
Continous optimization
87
Initial estimate for mean shapeThis is true, but misleading
88
Initial estimate for mean shape
True initial estimate: only the topology is really important.But the easiest way to get the topology is to build a rough template.89
Initial estimate for mean shape
True initial estimate: only the topology is really important.But the easiest way to get the topology is to build a rough template.90
91
Example results92
Example results93
Optimization
94
Number of images95
81632
Parameter sensitivityPixel terms: noise level paramsDimensionless terms
Smoothness terms
96
96
97
98
99
100
101
102Fitting Surface Models to Data
(^^)
@sumisumith
103