View
216
Download
0
Category
Preview:
Citation preview
3D photography
Kevin Köser, Marc Pollefeys
Spring 2012
http://www.cvg.ethz.ch/teaching/2012spring/3dphoto/
Feb 20 Introduction
Feb 27 Lecture: Geometry, Camera Model, Calibration
Mar 5 Lecture: Features, Tracking/Matching
Mar 12 Project Proposals by Students
Mar 19 Lecture: Epipolar Geometry
Mar 26 Short lecture “Stereo”
+ 2 papers (2 teams of 2 students)
Apr 2 Short lecture “Active Ranging, Structured Light” + 2 papers
Apr 9 Easter
Apr 16 Project Updates (Sechseläuten in afternoon)
Apr 23 Short lecture “Volumetric Modeling” +2 papers
Apr 30 Short lecture “Single View” + 2 papers
May 7 Short lecture “SfM from Photo Collections” + 2 papers
May 14 Short lecture “Silhouettes” + 2 papers
May 21 Short lecture + 2 papers
May 28 Pentecost / White Monday
May 29-June 1 Final Demos
Schedule (tentative)
Projective Geometry and Camera model
Class 2
points, lines, planes conics and quadrics
transformations camera model
See http://www.cs.unc.edu/~marc/tutorial/
Book: Hartley/Zisserman: “Multiple View Geometry in Computer Vision”
Homogeneous coordinates
0=++ cbyax ( ) ( ) 0=1x,y,a,b,cT
( ) ( ) 0≠∀,~ ka,b,cka,b,cTT
Homogeneous representation of 2D points and lines
equivalence class of vectors, any vector is representative
Set of all equivalence classes in R3(0,0,0)T forms P2
( ) ( ) 0≠∀,1,,~1,, kyxkyxTT
The point x lies on the line l if and only if
Homogeneous coordinates
Inhomogeneous coordinates ( )Tyx,
( )T321 ,, xxx but only 2DOF
Note that scale is unimportant for incidence relation
0=xlT
Points from lines and vice-versa
l'×l=x
Intersections of lines
The intersection of two lines and is l l'
Line joining two points
The line through two points and is x'×x=lx x'
Example
1=x
1=y
1
)1-,1,0( y
x
1
)1-,0,1( y
x
Note:
[ ] 'xx='x×x ×
with
0-
0-
-0
x
xy
xz
yz
Ideal points and the line at infinity
T0,,l'l ab
Intersections of parallel lines
( ) ( )TTand ',,=l' ,,=l cbacba
Example
1=x 2=x
Ideal points ( )T0,, 21 xx
Line at infinity ( )T1,0,0=l∞
∞22 l∪= RP
tangent vector
normal direction
-ab,
( )ba,
Note that in P2 there is no distinction
between ideal points and others
3D points and planes
Homogeneous representation of 3D points and planes
The point X lies on the plane π if and only if
0=π+π+π+π 44332211 XXXX
0=XπT
The plane π goes through the point X if and only if
0=XπT
Planes from points
0π
X
X
X
3
2
1
T
T
T
0=πX 0=πX 0,=πX π 321
TTTandfromSolve
(solve as right nullspace of ) π
T
T
T
3
2
1
X
X
X
Points from planes
0X
π
π
π
3
2
1
T
T
T
x=X M 321 XXXM
0=π MT
0=Xπ 0=Xπ 0,=Xπ X 321
TTTandfromSolve
(solve as right nullspace of ) X
T
T
T
3
2
1
π
π
π
Representing a plane by its span
Lines
T
T
B
AW μBλA
2×2
** 0=WW=WWTT
0001
1000W
0010
0100W*
Example: X-axis
(4dof)
Representing a line by its span
T
T
Q
PW*
μQλP
Dual representation (planes)
(Alternative: Plücker representation, details see e.g. H&Z)
Points, lines and planes
TX
WM 0=πM
Tπ
W*
M 0X M
W
X
*W
π
Common plane for point X and line W
Intersection line W and plane
Plücker coordinates
Elegant representation for 3D lines
jijiij ABBAl -
TL 344223141312 ,,,,, llllll 5∈P
0B,AB,A,detˆ| LL
( ) 0=|LL (Plücker internal constraint)
(two lines intersect)
( )123413421423231442133412ˆ+ˆ+ˆ+ˆ+ˆ+ˆ=ˆ| llllllllllllLL
(for more details see e.g. H&Z)
(with A and B points)
Conics
Curve described by 2nd-degree equation in the plane
0=+++++ 22 feydxcybxyax
0=+++++ 2
33231
2
221
2
1 fxxexxdxcxxbxax
3
2
3
1 ,x
xy
xx
x or homogenized
0=xx CT
or in matrix form
fed
ecb
dba
2/2/
2/2/
2/2/
Cwith
{ }fedcba :::::5DOF:
Five points define a conic
For each point the conic passes through
0=+++++ 22 feydxcyybxax iiiiii
or
( ) 0=.1,,,,, 22ciiiiii yxyyxx ( )T
fedcba ,,,,,=c
0
1
1
1
1
1
55
2
555
2
5
44
2
444
2
4
33
2
333
2
3
22
2
222
2
2
11
2
111
2
1
c
yxyyxx
yxyyxx
yxyyxx
yxyyxx
yxyyxx
stacking constraints yields
Dual conics
0=ll *C
TA line tangent to the conic C satisfies
Dual conics = line conics = conic envelopes
1-* CC In general (C full rank):
Degenerate conics
A conic is degenerate if matrix C is not of full rank
TT ml+lm=C
e.g. two lines (rank 2)
e.g. repeated line (rank 1)
Tll=C
l
l
m
Degenerate line conics: 2 points (rank 2), double point (rank1)
( ) CC ≠**Note that for degenerate conics
Quadrics and dual quadrics
(Q : 4x4 symmetric matrix) 0=QXXT
• 9 d.o.f.
• in general 9 points define quadric
• det Q=0 ↔ degenerate quadric
• tangent plane
Q
QX=π
0=πQπ *T
-1* Q=Q• relation to quadric (non-degenerate)
2D projective transformations
A projectivity is an invertible mapping h from P2 to itself
such that three points x1,x2,x3 lie on the same line if and
only if h(x1),h(x2),h(x3) do.
Definition:
A mapping h:P2P2 is a projectivity if and only if there
exist a non-singular 3x3 matrix H such that for any point
in P2 reprented by a vector x it is true that h(x)=Hx
Theorem:
Definition: Projective transformation
3
2
1
333231
232221
131211
3
2
1
'
'
'
x
x
x
hhh
hhh
hhh
x
x
x
x=x' Hor
8DOF
projectivity=collineation=projective transformation=homography
Transformation of 2D points, lines and conics
Transformation for lines
l=l' -TH
Transformation for conics
-1-TCHHC ='
Transformation for dual conics
THHCC
** ='
x=x' HFor a point transformation
Fixed points and lines
e=e λH
(eigenvectors H =fixed points)
l=l λTH
(eigenvectors H-T =fixed lines)
(1=2 pointwise fixed line)
Hierarchy of 2D transformations
100
2221
1211
y
x
taa
taa
100
2221
1211
y
x
tsrsr
tsrsr
333231
232221
131211
hhh
hhh
hhh
100
2221
1211
y
x
trr
trr
Projective
8dof
Affine
6dof
Similarity
4dof
Euclidean
3dof
Concurrency, collinearity,
order of contact (intersection,
tangency, inflection, etc.),
cross ratio
Parallellism, ratio of areas,
ratio of lengths on parallel
lines (e.g midpoints), linear
combinations of vectors
(centroids).
The line at infinity l∞
Ratios of lengths, angles.
The circular points I,J
lengths, areas.
invariants transformed
squares
The line at infinity
3
2
1
1
-
1t-
0ll
l
l
l
AA
AH
T-T
The line at infinity l is a fixed line under a projective transformation H if and only if H is an affinity
110
t2
1
x
x
xx A
AH
10
t11
1- AAHA
An affinity maps points
Lines map with an inverse, transposed H:
∞1∞-
∞ l
1
0
0
1t-
0ll
A
AH
T-T
A
The line at infinity thus is mapped as:
The circular points
0
1
I i
0
1
J i
I
0
1
0
1
100
cossin
sincos
II
iseitss
tssi
y
x
S
H
The circular points I, J are fixed points under the projective transformation H iff H is a similarity
The circular points
“circular points”
0=++++ 2
33231
2
2
2
1 fxxexxdxxx 0=+ 2
2
2
1 xx
l∞
T
T
0,-,1J
0,,1I
i
i
( ) ( )TT0,1,0+0,0,1=I i
Algebraically, encodes orthogonal directions
03 x
Conic dual to the circular points
TT JIIJ*
∞ C
000
010
001*
∞C
T
SS HCHC*
∞*
∞
The dual conic is fixed conic under the
projective transformation H iff H is a similarity
*
∞C
l∞
I
J
Angles
( )( )2
2
2
1
2
2
2
1
2211
++
+=cos
mmll
mlmlθ
( )T321 ,,=l lll ( )T
321 ,,=m mmm
Euclidean:
Projective: mmll
mlcos
**
*
CC
C
TT
T
0=ml *
∞CT
(orthogonal)
Transformation of 3D points, planes and quadrics
Transformation for lines
( )l=l' -TH
Transformation for conics
( )-1-TCHHC ='
Transformation for dual conics
( )THHCC
** ='
( )x=x' H
For a point transformation
X=X' H
π=π' -TH
-1-TQHH=Q'
THHQ=Q' **
(cfr. 2D equivalent)
Hierarchy of 3D transformations
vTv
tAProjective
15dof
Affine
12dof
Similarity
7dof
Euclidean
6dof
Intersection and tangency
Parallellism of planes,
Volume ratios, centroids,
The plane at infinity π∞
Angles, ratios of length
The absolute conic Ω∞
Volume
10
tAT
10
tRT
s
10
tRT
The plane at infinity
∞∞∞ π
1
0
0
0
1t-
0ππ
A
AH
TT
A
The plane at infinity π is a fixed plane under a projective transformation H iff H is an affinity
1. canonical position
2. contains directions
3. two planes are parallel line of intersection in π∞
4. line // line (or plane) point of intersection in π∞
( )T1,0,0,0=π∞
( )T0,,,=D 321 XXX
The absolute conic
The absolute conic Ω∞ is a fixed conic under the projective transformation H iff H is a similarity
04
2
3
2
2
2
1
X
XXX
The absolute conic Ω∞ is a (point) conic on π.
In a metric frame:
T321321 ,,I,, XXXXXXor conic for directions:
(with no real points)
1. Ω∞ is only fixed as a set
2. Circle intersect Ω∞ in two circular points
3. Spheres intersect π∞ in Ω∞
The absolute dual quadric
00
0I*
∞ T
The absolute dual quadric Ω*∞ is a fixed conic under
the projective transformation H iff H is a similarity
1. plane at infinity π∞ is the nullvector of Ω∞
2. Angles:
( )( )2
*
∞21
*
∞1
2
*
∞1
πΩππΩπ
πΩπ=cos
TT
T
θ
Pinhole camera model
TT ZfYZfXZYX )/,/(),,(
101
0
0
1
Z
Y
X
f
f
Z
fY
fX
Z
Y
X
linear projection in homogeneous coordinates!
Principal point offset
T
yx
T pZfYpZfXZYX )+/,+/(),,(
principal point T
yx pp ),(
101
0
0
1
Z
Y
X
pf
pf
Z
ZpfY
ZpfX
Z
Y
X
y
x
x
x
Principal point offset
101
0
0
Z
Y
X
pf
pf
Z
ZpfY
ZpfX
y
x
x
x
camX0|IKx
1
y
x
pf
pf
K calibration matrix
Camera rotation and translation
( )C~
-X~
R=X~
cam
X10
RC-R
1
10
C~
R-RXcam
Z
Y
X
camX0|IKx XC~
-|IKRx
t|RKP C~
R-t PX=x
~
General projective camera
1
yx
xx
p
ps
K
1
yx
xx
p
p
K
C~
|IKRP
non-singular
11 dof (5+3+3)
t|RKP
intrinsic camera parameters
extrinsic camera parameters
Radial distortion
• Due to spherical lenses (cheap)
• Model:
R
y
xyxKyxKyx ...))()(1(),( 44
2
22
1R
http://foto.hut.fi/opetus/260/luennot/11/atkinson_6-11_radial_distortion_zoom_lenses.jpg straight lines are not straight anymore
Projector model
Relation between pixels and rays in space
(dual of camera)
(main geometric difference is vertical principal point offset
to reduce keystone effect)
?
Action of projective camera on points
and lines
forward projection of line
( ) μb+a=μPB+PA=μB)+P(A=μX
back-projection of line
lP=Π T
PXlX TT ( )PX= x0;=xlT
PX=x
projection of point
Action of projective camera on conics
and quadrics
back-projection to cone
CPPQ T
co 0=CPXPX=Cxx TTT
( )PX=x
projection of quadric
TPPQC ** 0lPPQlQ T*T*T
( )lP=Π T
Practical Camera Calibration
Unknown: constant camera intrinsics K
(varying) camera pose R,t
Known: 3D coordinates of chessboard corners
=> Define to be the z=0 plane ( X=[X1 X2 0 1]’ )
Point is mapped as λx = K (r1 r2 r3 t) X
x = K (r1 r2 t) [X1 X2 1]’
1
yy
xx
pf
pf
K
Homography H between image and chess coordinates,
estimate from known Xi and measured xi
Method and Pictures from Zhang (ICCV’99): “Flexible Camera Calibration By Viewing a
Plane From Unknown Orientations”
Direct Linear Transformation (DLT)
ii Hxx 0Hxx ii
i
i
i
i
xh
xh
xh
Hx3
2
1
T
T
T
iiii
iiii
iiii
ii
yx
xw
wy
xhxh
xhxh
xhxh
Hxx12
31
23
TT
TT
TT
0
h
h
h
0xx
x0x
xx0
3
2
1
TTT
TTT
TTT
iiii
iiii
iiii
xy
xw
yw
Tiiii wyx ,,x
0hA i
Direct Linear Transformation (DLT)
• Equations are linear in h
0
h
h
h
0xx
x0x
xx0
3
2
1
TTT
TTT
TTT
iiii
iiii
iiii
xy
xw
yw
0AAA 321 iiiiii wyx
0hA i
• Only 2 out of 3 are linearly independent
(indeed, 2 eq/pt)
0
h
h
h
x0x
xx0
3
2
1
TTT
TTT
iiii
iiii
xw
yw
(only drop third row if wi’≠0)
• Holds for any homogeneous
representation, e.g. (xi’,yi’,1)
Direct Linear Transformation (DLT)
• Solving for H
0Ah 0h
A
A
A
A
4
3
2
1
size A is 8x9 or 12x9, but rank 8
Trivial solution is h=09T is not interesting
1-D null-space yields solution of interest
pick for example the one with 1h
Direct Linear Transformation (DLT)
• Over-determined solution
No exact solution because of inexact measurement
i.e. “noise”
0Ah 0h
A
A
A
n
2
1
Find approximate solution
- Additional constraint needed to avoid 0, e.g.
- not possible, so minimize
1h
Ah0Ah
DLT algorithm
Objective
Given n≥4 2D to 2D point correspondences {xi↔xi’},
determine the 2D homography matrix H such that xi’=Hxi
Algorithm
(i) For each correspondence xi ↔xi’ compute Ai. Usually
only two first rows needed.
(ii) Assemble n 2x9 matrices Ai into a single 2nx9 matrix A
(iii) Obtain SVD of A. Solution for h is last column of V
(iv) Determine H from h
Importance of normalization
0
h
h
h
0001
1000
3
2
1
iiiiiii
iiiiiii
xyxxxyx
yyyxyyx
~102 ~102 ~102 ~102 ~104 ~104 ~102 1 1
orders of magnitude difference!
Monte Carlo simulation
for identity computation based on 5 points
(not normalized ↔ normalized)
Normalized DLT algorithm
Objective
Given n≥4 2D to 2D point correspondences {xi↔xi’},
determine the 2D homography matrix H such that xi’=Hxi
Algorithm
(i) Normalize points
(ii) Apply DLT algorithm to
(iii) Denormalize solution
,x~x~ ii
inormiinormi xTx~,xTx~
norm
-1
norm TH~
TH
1
norm
100
2/0
2/0
T
hhw
whw
Geometric distance measured coordinates
estimated coordinates
true coordinates
xxx
2H
xH,xargminH ii
i
d
Error in one image
e.g. calibration pattern
221-
H
Hx,xxH,xargminH iiii
i
dd
Symmetric transfer error
d(.,.) Euclidean distance (in image)
ii
iiii
i
ii ddii
xHx subject to
x,xx,xargminx,x,H22
x,xH,
Reprojection error
Statistical cost function and Maximum Likelihood Estimation
• Optimal cost function related to noise model
• Assume zero-mean isotropic Gaussian noise (assume outliers removed)
22
2/xx,
2πσ2
1xPr de
22
i 2xH,x /
2iπσ2
1H|xPr
ide
i
constantxH,xH|xPrlog2
i2iσ2
1id
Error in one image
Maximum Likelihood Estimate
2
i xH,x id
Statistical cost function and Maximum Likelihood Estimation
• Optimal cost function related to noise model
• Assume zero-mean isotropic Gaussian noise (assume outliers removed)
22
2/xx,
2πσ2
1xPr de
22
i2
i 2xH,xx,x /
2iπσ2
1H|xPr
ii dd
ei
Error in both images
Maximum Likelihood Estimate
2i
2
i x,xx,x ii dd
Gold Standard algorithm
Objective
Given n≥4 2D to 2D point correspondences {xi↔xi’},
determine the Maximum Likelyhood Estimation of H
(this also implies computing optimal xi’=Hxi)
Algorithm
(i) Initialization: compute an initial estimate using
normalized DLT or RANSAC
(ii) Geometric minimization of reprojection error:
● Minimize using Levenberg-Marquardt over 9 entries of h
or Gold Standard error:
● compute initial estimate for optimal {xi}
● minimize cost over {H,x1,x2,…,xn}
● if many points, use sparse method
2i
2
i x,xx,x ii dd
Practical Camera Calibration
Unknown: constant camera intrinsics K
(varying) camera pose R,t
Known: 3D coordinates of chessboard corners
=> Define to be the z=0 plane ( X=[X1 X2 0 1]’ )
Point is mapped as λx = K (r1 r2 r3 t) X
x = K (r1 r2 t) [X1 X2 1]’
1
yy
xx
pf
pf
K
Homography H between image and chess coordinates,
estimate from known Xi and measured xi
Method and Pictures from Zhang (ICCV’99): “Flexible Camera Calibration By Viewing a
Plane From Unknown Orientations”
Practical Camera Calibration
1
yy
xx
pf
pf
K
H has the shape:
H = λK(r1 r2 t)
<=> K−1H = λ(r1 r2 t)
With rotation matrix constraints
r1Tr2 = 0
r1Tr1=r2
Tr2=1
„Squaring“ yields 2 constraints on IAC (KKT)-1
h1T K−TK−1h2 = 0
hT1K
−TK−1h1 = hT2K
−TK−1h2
n images provide 2n constraints on IAC
More measurements => better estimate of IAC !
Practical Camera Calibration
1
yy
xx
pf
pf
K
Given an estimate for the IAC, factorize (KKT)-1
Factorize
[Obtain Poses]
Clamp
Optimize
Recommended