80
Lecture 7 - Silvio Savarese 3-Feb-15 • Active stereo • Structured lighting • Depth sensing • Volumetric stereo: • Space carving • Shadow carving • Voxel coloring Lecture 8 Active stereo & Volumetric stereo Reading: [Szelisky] Chapter 11 “Multiview stereo” Seitz, S. M., & Dyer, C. R. (1999). Photorealistic scene reconstruction by voxel coloring.International Journal of Computer Vision, 35(2), 151-173. S. Savarese, M. Andreetto, H. Rushmeier, F. Bernardini and P. Perona, 3D Reconstruction by Shadow Carving: Theory and Practical Evaluation, International Journal of Computer Vision (IJCV) , 71(3), 305-336, 2006

Lecture 8 Active stereo &

Embed Size (px)

Citation preview

Page 1: Lecture 8 Active stereo &

Lecture 7 - Silvio Savarese 3-Feb-15

• Active  stereo•  Structured  lighting•  Depth  sensing

• Volumetric  stereo:• Space  carving• Shadow  carving• Voxel  coloring

Lecture  8Active  stereo  &    Volumetric  stereo

Reading:      [Szelisky]  Chapter  11  “Multi-­‐view  stereo”    

Seitz, S. M., & Dyer, C. R. (1999). Photorealistic scene reconstruction by voxel coloring.International Journal of Computer Vision, 35(2), 151-173.

S. Savarese, M. Andreetto, H. Rushmeier, F. Bernardini and P. Perona, 3D Reconstruction by Shadow Carving: Theory and Practical Evaluation, International Journal of Computer Vision (IJCV) , 71(3), 305-336, 2006

Page 2: Lecture 8 Active stereo &

Traditional stereo

What’s the main problem in traditional stereo?We need to find correspondences!

P

Camera O2

p’

O1

p

Camera

Page 3: Lecture 8 Active stereo &

Active stereo (point)

Replace one of the two cameras by a projector- Single camera - Projector geometry calibrated- What’s the advantage of having the projector?

P

O2

p’

Correspondence problem solved!

Camera

p

Projector (e.g. DLP)

image

projector virtual image

Page 4: Lecture 8 Active stereo &

Active stereo (stripe)

Projector (e.g. DLP) O

-Projector and camera are parallel- Correspondence problem solved!

Camera

p’p

P

projector virtual image image

l

L

l’

Page 5: Lecture 8 Active stereo &

Calibrating the system

Projector (e.g. DLP)O

- Use calibration rig to calibrate camera and localize rig in 3D- Project stripes on rig and calibrate projector

Camera

projector virtual image image

Page 6: Lecture 8 Active stereo &

Laser scanning

• Optical triangulation– Project a single stripe of laser light– Scan it across the surface of the object– This is a very precise version of structured light scanning

Digital Michelangelo Projecthttp://graphics.stanford.edu/projects/mich/

Source: S. Seitz

Page 7: Lecture 8 Active stereo &

The Digital Michelangelo Project, Levoy et al.

Laser scanning

Source: S. Seitz

Page 8: Lecture 8 Active stereo &

Light source O

Active stereo (shadows) J. Bouguet & P. Perona, 99

- 1 camera,1 light source- very cheap setup- calibrated the light source

Page 9: Lecture 8 Active stereo &

Active stereo (shadows)

Page 10: Lecture 8 Active stereo &

Active stereo (color-coded stripes)

- Dense reconstruction - Correspondence problem again- Get around it by using color codes

projectorO

L. Zhang, B. Curless, and S. M. Seitz 2002S. Rusinkiewicz & Levoy 2002

LCD or DLP displays

Page 11: Lecture 8 Active stereo &

L. Zhang, B. Curless, and S. M. Seitz. Rapid Shape Acquisition Using Color Structured Light and Multi-pass Dynamic Programming. 3DPVT 2002

Rapid shape acquisition: Projector + stereo cameras

Page 12: Lecture 8 Active stereo &

Pattern of projected infrared points to generate a dense 3D image

Active stereo – the kinect sensor

• Infrared laser projector combined with a CMOS sensor• Captures video data in 3D under any ambient light conditions.

Depth map

Source: wikipedia

Page 13: Lecture 8 Active stereo &

Lecture 7 - Silvio Savarese 3-Feb-15

Lecture  8Active  stereo  &    Volumetric  stereo • Active  stereo

•  Structured  lighting•  Depth  sensing

• Volumetric  stereo:• Space  carving• Shadow  carving• Voxel  coloring

Page 14: Lecture 8 Active stereo &

“Traditional” Stereo

Goal: estimate the position of P given the observation ofP from two view pointsAssumptions: known camera parameters and position (K, R, T)

p’

P

p

O1 O2

P

Page 15: Lecture 8 Active stereo &

Subgoals: 1. Solve the correspondence problem2. Use corresponding observations to triangulate

“Traditional” Stereo

p’

P

p

O1 O2

P

Page 16: Lecture 8 Active stereo &

Volumetric stereo

1. Hypothesis: pick up a point within the volume

O1O2

Working volume

2. Project this point into 2 (or more) images

Assumptions: known camera parameters and position (K, R, T)

3. Validation: are the observations consistent?

P

Page 17: Lecture 8 Active stereo &

Consistency based on cues such as:

- Contours/silhouettes à Space carving- Shadows à Shadow carving- Colors à Voxel coloring

Page 18: Lecture 8 Active stereo &

Lecture 7 - Silvio Savarese 3-Feb-15

Lecture  8Active  stereo  &    Volumetric  stereo

Reading:      [Szelisky]  Chapter  11  “Multi-­‐view  stereo”    

• Active  stereo•  Structured  lighting•  Depth  sensing

• Volumetric  stereo:• Space  carving• Shadow  carving• Voxel  coloring

Page 19: Lecture 8 Active stereo &

Contours/silhouettes

• Contours  are  a  rich  source  of    geometric  information  

Page 20: Lecture 8 Active stereo &

Apparent  Contour• Projection  of  the  locus  of  points  on  the  surface  which  separate  

the  visible  and  occluded  parts  on  the  surface[sato  &  cipolla]

Image

Camera

apparent contour

Page 21: Lecture 8 Active stereo &

Silhouettes

Image

Camera

silhouette

A silhouette is defined as the area enclosed by the apparent contours

Page 22: Lecture 8 Active stereo &

Detecting  silhouettes

Image

Camera

Object

Easy contour segmentation

Page 23: Lecture 8 Active stereo &

Detecting  silhouettes

Page 24: Lecture 8 Active stereo &

How  can  we  use  contours?

Image

Camera

Object

Objectapparent contour

Visual cone

Camera Center

Silhouette

Page 25: Lecture 8 Active stereo &

Image Camera

Object

Objectapparent contour

Visual cone

2D slice:Camera

Image line

Object

Visual cone

Object’s silhouette

How  can  we  use  contours?

Silhouette

Page 26: Lecture 8 Active stereo &

View point 1

Object

View point 2

Object

Object estimate(visual hull)

The views are calibrated

How  can  we  use  contours?

Page 27: Lecture 8 Active stereo &

How  to  perform  visual  cones  intersection?• Decompose  visual  cone  in  polygonal  surfaces          (among  others:  Reed  and  Allen  ‘99)

Page 28: Lecture 8 Active stereo &

Space  carving

• Using   contours/silhouettes   in   volumetric   stereo,   also   called  space  carving

[ Martin and Aggarwal (1983) ]

voxel

Page 29: Lecture 8 Active stereo &

Computing  Visual  Hull  in  2D

Page 30: Lecture 8 Active stereo &

Computing  Visual  Hull  in  2D

Page 31: Lecture 8 Active stereo &

Computing  Visual  Hull  in  2D

Page 32: Lecture 8 Active stereo &

Visual  hull:  an  upper  bound  estimate

Consistency:A voxel must be projected into a silhouette in each image

Computing  Visual  Hull  in  2D

Page 33: Lecture 8 Active stereo &

Space  carving  has  complexity  …

O(N3)

N

N

N

Octrees  (Szeliski    ’93)

Page 34: Lecture 8 Active stereo &

Complexity  Reduction:  Octrees

1

1

1

Page 35: Lecture 8 Active stereo &

• Subdiving  volume  in  sub-­‐volumes  of  progressive  smaller  size

2

2

2

Complexity  Reduction:  Octrees

Page 36: Lecture 8 Active stereo &

44

4

Complexity  Reduction:  Octrees

Page 37: Lecture 8 Active stereo &

Complexity reduction: 2D example

4 elements analyzed in this step

Page 38: Lecture 8 Active stereo &

16 elements analyzed in this step

Complexity reduction: 2D example

Page 39: Lecture 8 Active stereo &

52 elements are analyzed in this step

Complexity reduction: 2D example

Page 40: Lecture 8 Active stereo &

1+ 4 + 16 + 52 + 34x16 = 617 voxels have been analyzed in total

(rather than 32x32 = 1024)

16x34 voxels are analyzed in this step

Complexity reduction: 2D example

Page 41: Lecture 8 Active stereo &

Advantages  of  Space  carving

• Robust  and  simple• No  need  to  solve  for  correspondences

Page 42: Lecture 8 Active stereo &

Limitations  of  Space  carving

•  Accuracy  function  of  number              of  views

Not a good estimate

What else?

Page 43: Lecture 8 Active stereo &

• Concavities  are  not  modeled Concavity

For 3D objects:Are all types of concavities

problematic?

Limitations  of  Space  carving

Page 44: Lecture 8 Active stereo &

• Concavities  are  not  modeled Concavity

(hyperbolic regions are ok)

Visual hull

-­‐-­‐  see  Laurentini  (1995)

Limitations  of  Space  carving

Page 45: Lecture 8 Active stereo &

Space  carving:  A  Classic  Setup

Camera

Object

Turntable

Courtesy of Seitz & Dyer

Page 46: Lecture 8 Active stereo &

Space  carving:  A  Classic  Setup

Page 47: Lecture 8 Active stereo &

30 cm

24 poses (15O)voxel size = 1mm

Space  carving:  Experiments

Page 48: Lecture 8 Active stereo &

Space  carving:  Experiments

10cm

24 poses (15O)

voxel size = 2mm

6cm

Page 49: Lecture 8 Active stereo &

Space  carving:  Conclusions

• Robust  • Produce  conservative  estimates• Concavities  can  be  a  problem• Low-­‐end  commercial  3D  scanners

blue backdrop

turntable

Page 50: Lecture 8 Active stereo &

Lecture 7 - Silvio Savarese 3-Feb-15

Lecture  8Active  stereo  &    Volumetric  stereo • Active  stereo

•  Structured  lighting•  Depth  sensing

• Volumetric  stereo:• Space  carving• Shadow  carving• Voxel  coloring

Page 51: Lecture 8 Active stereo &

Shape  from  Shadows• Self-­‐shadows    are  visual  cues  for  shape  recovery

Self-shadows indicate concavities (no modeled by contours)

Page 52: Lecture 8 Active stereo &

Shadow  carving:  The  Setup

Camera

Turntable

Object

Array of lights

[Savarese et al. 2001]

Page 53: Lecture 8 Active stereo &

[Savarese et al ’01]

Array of lights

Camera

Turntable

Object

Shadow  carving:  The  Setup

Page 54: Lecture 8 Active stereo &

Shadow  carving:  The  Setup

Array of lights

Camera

Turntable

Object

[Savarese et al ’01]

Page 55: Lecture 8 Active stereo &

Shadow  carving

+

Object’s upper bound

Self-shadows

[Savarese et al ’01]

Page 56: Lecture 8 Active stereo &

Algorithm:  Step  k

Object

Light source

Camera

Image line

Image shadow

Upper-bound from step k-1

Image

Page 57: Lecture 8 Active stereo &

Image

Object

Light source

Camera

Image line

Image shadow

Upper-bound from step k-1

Light source

virtual image shadow

?

Algorithm:  Step  k

Theorem:A voxel that projects into an image shadow AND an virtual image shadow cannot belong to the object.

projection of the shadow points in the image to the upper bound estimate of the object

virtual light image

Page 58: Lecture 8 Active stereo &

Image

Light source

Image line

Image shadow

Upper-bound from step k-1

Object

Light source

Camera

Virtual image shadow

No further volumecan be removedCarving process always conservative

Proof of correctness Consistency:

In order for a voxel to be removed it must project into both image shadow and virtual image shadow

Properties: virtual light image

Algorithm:  Step  k

Page 59: Lecture 8 Active stereo &

Image

Light source

Image line

Image shadow

Upper-bound from step k-1

Object

Light source

Camera

Complexity?O(2N3)

Algorithm:  Step  k

Virtual image shadow

virtual light image

Page 60: Lecture 8 Active stereo &

Simulating  the  System

- 24 positions- 4 lights

- 72 positions- 8 lights

Page 61: Lecture 8 Active stereo &

- 16 positions- 4 lights

Results

Space carving

Shadow carving

Page 62: Lecture 8 Active stereo &

Results

Space carving Shadow carving

Page 63: Lecture 8 Active stereo &

Shadow  carving:  Summary• Produces  a  conservative  volume  estimate• Accuracy  depending  on  view  point  and  light  source  number• Limitations  with  reflective    &  low  albedo  regions

Page 64: Lecture 8 Active stereo &

Lecture 7 - Silvio Savarese 3-Feb-15

Lecture  8Active  stereo  &    Volumetric  stereo • Active  stereo

•  Structured  lighting•  Depth  sensing

• Volumetric  stereo:• Space  carving• Shadow  carving• Voxel  coloring

Page 65: Lecture 8 Active stereo &

Voxel  Coloring[Seitz & Dyer (‘97)]

[R. Collins (Space Sweep, ’96)]

• Color/photo-­‐consistency  • Jointly  model  structure  

and  appearance

Page 66: Lecture 8 Active stereo &

Basic  Idea

View1View3

View2

Is this voxel in or out?

Page 67: Lecture 8 Active stereo &

View1View3

View2

Basic  Idea

Page 68: Lecture 8 Active stereo &

UniquenessFr

om S

eitz

& D

yer (

‘97)

observations

One possible solution

Page 69: Lecture 8 Active stereo &

• Multiple  consistent  scenes

How  to  fix  this?Need   to   use   a   visibility  constraint

Uniqueness

Page 70: Lecture 8 Active stereo &

The  Algorithm

C

Page 71: Lecture 8 Active stereo &

Algorithm  Complexity

• Voxel  coloring  visits  each  N3  voxels  only  once  • Project  each  voxel  into  L  images

à O(L  N3)

NOTE:  not  function  of  the  number  of  colors

Page 72: Lecture 8 Active stereo &

Photoconsistency  Test

If C > λ= threshold ➔ voxel consistent

C = corr ( , )

v1 v2

Page 73: Lecture 8 Active stereo &

A  Critical  Assumption:  Lambertian  Surfaces

I1(p) I2(p)I1(p) = I2(p)

Page 74: Lecture 8 Active stereo &

Non  Lambertian  Surfaces

Page 75: Lecture 8 Active stereo &

Experimental  Results

Dinosaur 72 k voxels colored7.6 M voxels tested7 min to compute on a 250MHz

Image  source:    http://www.cs.cmu.edu/~seitz/vcolor.html

Page 76: Lecture 8 Active stereo &

Flower 70 k voxels colored7.6 M voxels tested7 min to compute on a 250MHz

Image  source:    http://www.cs.cmu.edu/~seitz/vcolor.html

Experimental  Results

Page 77: Lecture 8 Active stereo &

Image  source:    http://www.cs.cmu.edu/~seitz/vcolor.html

Room + weird people

Experimental  Results

Page 78: Lecture 8 Active stereo &

Voxel  Coloring:  Conclusions

• Good  things– Model  intrinsic  scene  colors  and  texture– No  assumptions  on  scene  topology

• Limitations:– Constrained  camera  positions– Lambertian  assumption

Page 79: Lecture 8 Active stereo &

Further  Contributions

• A  Theory  of  Space  carving– Voxel  coloring  in  more  general  framework– No  restrictions  on  camera  position

• Probabilistic  Space  carving

[Kutulakos & Seitz ’99]

[Broadhurst & Cipolla, ICCV 2001][Bhotika, Kutulakos et. al, ECCV 2002]

Page 80: Lecture 8 Active stereo &

Next lecture…

Fitting and Matching