Upload
presta
View
47
Download
3
Embed Size (px)
DESCRIPTION
3D Modeling using Multi-camera and Multi-lighting (MVML) Dome. Broadband Network and Digital Media Lab (BBNC), Automation Department, Tsinghua University Yebin Liu http://media.au.tsinghua.edu.cn/liuyebin.jsp. Introduction. 宽带网数字媒体技术实验室. Broadband Network & Digital Media Lab. - PowerPoint PPT Presentation
Citation preview
3D Modeling using Multi-camera and Multi-lighting (MVML) Dome
Broadband Network and Digital Media Lab (BBNC),Automation Department, Tsinghua University
Yebin Liuhttp://media.au.tsinghua.edu.cn/liuyebin.jsp
13D Modeling Using MVML Dome
23/4/21
23/4/212 3D Modeling Using MVML Dome
Introduction宽带网数字媒体技术实验室
10-year history 1999~2009
1. finished or undergoing more than 10 national major R/D projects2. Awards for National Scientific and Technological Progress
Broadband Network & Digital Media Lab
Introduction
23/4/213 3D Modeling Using MVML Dome
宽带网数字媒体技术实验室Broadband Network & Digital Media Lab
• Born in Dec,1964. Ph.D, IEEE Senior Member • Distinguished Young Scholar of NSF China • Second Prize for National Scientific and Technological Progress 2008• The chief scientist of national 973 Program
Head: Prof. Qionghai Dai
16 PHD Students and 23 Master studentsMost of the PhDs are junior, and interest in Vision and Graphics
I spent 7 years in this lab. My working experiences: video streaming and coding (02’) light field and 3D display (05’) MVML dome (07’)
Homepage://media.au.tsinghua.edu.cn
23/4/214 3D Modeling Using MVML Dome
Contents
23/4/215 3D Modeling Using MVML Dome
Contents
Background
• 3D Studio– IBMR– FVV and performance capture
• Light Stage– IBL– Performance relighting
• Light Field– IBR– Computational photography
23/4/216 3D Modeling Using MVML Dome
23/4/217 3D Modeling Using MVML Dome
Multi-camera Multi-light Dome
Diameter 6m
40 flea2 camera
Lamp for color calibration
screen for chrome-keying
A Combination of 3D Studio, Light Stage and Light Field
LEDs for lighting
23/4/218 3D Modeling Using MVML Dome
Topology and the Control Module
9 3D Modeling Using MVML Dome 23/4/21
Multi-camera Multi-light Dome
10 3D Modeling Using MVML Dome 23/4/21
Multi-camera Multi-light Dome
Multi-camera Multi-light Dome
• Resolution: 1024 by 768, frame rate: 25fps• FVV data and MVML data are available on http://media.au.tsinghua.edu.cn/fvv.jsp http://media.au.tsinghua.edu.cn/mvml.jsp
12 3D Modeling Using MVML Dome 23/4/21
Multi-camera Multi-light Dome
23/4/2113 3D Modeling Using MVML Dome
Contents
Multi-view Stereo (MVS): reconstruct 3D model from multiple calibrated photographs of a realistic object
Background
Available MVS methods (according to Middlebury Benchmark):
MVS in category 3,4 are accurate but not robust
1. Volumetric based MVS2. Surface evolution MVS3. Depth Map merging MVS (our work is in this kind)4. Feature propagation MVS
14 3D Modeling Using MVML Dome 23/4/21
Former work: PCMVS• A Point Cloud (Depth Map) based FVV System1
23/4/2115 3D Modeling Using MVML Dome
Our work.
Furukawa’07.
FVV1
FVV2
Depth map based MVS achieve both high accuracy and robustness!
1. Y.Liu, Q.Dai, W.Xu. A Point cloud based Multi-view stereo for Free-viewpoint video . accepted in TVCG.
Former work: PCMVS
PCMVS-10 PCMVS-20 PCMVS-BINO
Under sparse array mode:
Binocular is much better
Stereo matching is crucial in MVS
23/4/2116 3D Modeling Using MVML Dome
Epipolar line
Target view Reference view
3D Surface
Matching Error!
2 2
( )( )
( ) ( )
i i
i i
p p q qNCC
p p q q
Why Stereo Matching Hard?
23/4/2117 3D Modeling Using MVML Dome
18
Popular Matching Techniques
3D Modeling Using MVML Dome 23/4/21
Optical flow Stereo Mview Stereo
Optimization mechanism Method cost define unit
We Propose :
Proposed Pipeline
193D Modeling Using MVML Dome
23/4/21
Variational
20
Variation Goes for MVS
3D Modeling Using MVML Dome 23/4/21
( ) ( ) ( )D SE w E w E w 2
2
( ( ) ( )( )
( ( ) ( )
r t
D D
r t
I d w IE w dxdy
I d w I
x x
x x
2( )S SE w w dxdy
1.Continuous and dense Capture
2.Rotation invariant matching
3.VH and epipolar reduce the probability of local minima
Multiple Multi-scale IterationMulti-scale coarse-to-fine iteration technique*
Multi-scale image pyramid
Multiple candidates by starting the iteration from different scales
21 3D Modeling Using MVML Dome 23/4/21
Depthmap v1 Depthmap v2
Depthmap v3Depthmap v4
*. T.Brox, A.Bruhn, N.Papenberg, and J.Weickert. High accuracy optical flow estimation based on a theory for warping. ECCV’04.
Multiple CandidatesHigher starting scale
Different
versions
generate
different
well
reconstruction
areas
well reconstruct
bad reconstruct
22 3D Modeling Using MVML Dome 23/4/21
Target view Reference view
Candidate patches
Accurate Matching
2 2
( )( )
( ) ( )
i i
i i
p p q qNCC
p p q q
Choose the best patch from all candidates using patch based NCC measurement
Refined Depth Map Synthesis
23/4/2123 3D Modeling Using MVML Dome
Multi-view Depth Maps Merging
23/4/2124 3D Modeling Using MVML Dome
View 1 before
View 1 after View 5 after
View 5 before View 9 before
View 9 after
Combine before
Combine after
MStereo Benchmark by Middlebury college
Measured by accuracy and completeness
Reconstruction Performance
25 3D Modeling Using MVML Dome 23/4/21
FurukawaCVPR07
High performance on detail reconstruction
LiuCVPR09
26 3D Modeling Using MVML Dome 23/4/21
BradleyCVPR07
Ground Truth
Experimental Results
27 3D Modeling Using MVML Dome 23/4/21
Multiple candidates + NCC metric on Continuous surface
Detail reconstruction
Robust reconstruction
CMVS Summary
28 3D Modeling Using MVML Dome 23/4/21
Continuous Depth EstimationPros:
Cons:
Requires an initial depth map for each viewVisual hull may not necessary
Complexity: MSS optical flow is the bottleneck
(half an hour for DinoSparseRing)Optimization the OF algorithm is important
23/4/2129 3D Modeling Using MVML Dome
Contents
Cons: Highlights, Textureless, Cycle-texture3D
Reconstruction
Multi-view Stereo Multi-view Single-light
Multi-view Multi-light 3D Reconstruction
Background
Photometric Stereo Multi-light Single-view
Multi-light Multi-view
Cons: Continuous assumptionOnly obtain shape, not geometry
30 3D Modeling Using MVML Dome 23/4/21
31 3D Modeling Using MVML Dome 23/4/21
Lights
Views
Directional Illumination
Multiplex Illumination
Multi-view Photometric Stereo
32 3D Modeling Using MVML Dome 23/4/21
• No Illumination Constraints - Illuminations is represented by low order spherical harmonics*
- No constraints on location, intensity and assumptions of the lights
*. R.Basri, D.W.Jacobs, "Lambertian Reflectance and Linear Subspaces", TPAMI'03
Multi-view Photometric Stereo
Multi-view Photometric Stereo
33 3D Modeling Using MVML Dome 23/4/21
• No Manual Interactions - Space-light MVS provides initial normal information
- Alternating optimization (normal and light)#
• Watertight and Robust Reconstruction - Multi-view images provide watertight reconstruction - Suitable for objects with small portion of non-Lambert. surface
• Key Model Recon. for Performance Capture Systems*
- Bypass laser scan and model registration - Standstill for 1 second
* E.d.Aguiar, C.Stoll, C.Theobalt, N.Ahmed, H.P.Seidel, S.Thrun, "Performance Capture from Sparse Multi-view Video", SIGGRAPH'08
* D.Vlasic, I.Baran, W.Matusik, J.Popovic, "Articulated Mesh Animation from Multi-view Silhouettes", SIGGRAPH'08
# R.Basri, D.Jacobs, I.Kemelmacher, "Photometric Stereo with General, Unknown Lighting", IJCV'06
Experimental Results
34 3D Modeling Using MVML Dome 23/4/21
Experimental Results
35 3D Modeling Using MVML Dome 23/4/21
36 3D Modeling Using MVML Dome 23/4/21
Experimental Results
Results: Impacts of Lights
23/4/2137 3D Modeling Using MVML Dome
Results: Synthetic Datasets
23/4/2138 3D Modeling Using MVML Dome
D.Vlasic, I.Baran, W.Matusik, J.Popovic, "Articulated Mesh Animation from Multi-view Silhouettes", SIGGRAPH'08
Results: Relighting
39 3D Modeling Using MVML Dome 23/4/21
Results: Relighting
23/4/2140 3D Modeling Using MVML Dome
1. How to extend to Motion Actors?2. Extend to Outdoor Scenes (Complex illumination)
3. MPS under Sparse Sampling
MPS SummaryNo illumination constraints (Unknown lights)
No manual interactions Watertight and robust reconstruction
Practical in performance capture, animation, relighting, etc.
23/4/2141 3D Modeling Using MVML Dome
Pros:
Future works:
√ √ √ √ √ √
√ √ √ √ √ √
√ √ √ √ √ √
√ √ √ √ √ √
light:
view
√ × √ √ √ ×
× √ × × √ √
√ × × √ × √
√ √ × √ √ √
light:
view
Contents
23/4/2142 3D Modeling Using MVML Dome
The Vision Field
23/4/2143 3D Modeling Using MVML Dome
Vision Field: the full space (1d View+1d Light+1d Time) on where vision problems define “Matching” in lower subspace vs. “Fusion” in higher subspace and full space
Vision
MVML dome is a sampling of the visual field
Vision Field: Future Works
• For free-motion object, It is impossible to capture the full Time-Light subspace.
• Only a few works have addressed this problem*
• Can be adopted in MPS and relighting motion objects.
23/4/2144 3D Modeling Using MVML Dome
√ √ √ √ √
√ √ √ √ √
√ √ √ √ √
√ √ √ √ √
light
time
√
√
√
√ √
light
timeImpossible mode
√ √ √ √ √
√ √ √ √ √
√ √ √ √ √
√ √ √ √ √
light
time1d Sparse sample
Matching and Completion
Reconstructed subspace
*. B.D.Decker, J.Kautz, T.Mertens, P.Bekaert, "Capturing Multiple Illumination Conditions using Time and Color Multiplexing", CVPR'09
23/4/2145 3D Modeling Using MVML Dome
1/30 S1/30 S 1/30 S1/30 S
Surface Matching
N Lights
Vision Field: Future Works
N Lights
Target Model
Texture Mapping
Tracking Model
Reconstruction of full space vision field using surface matching
Surface animation techniques:
Vision Field: Future Works
• The MVML dome achieves the samplings of the vision field.
• It is a foundation platform for the research on sampling theory of the vision field.
• We hope to extend results obtain in MVML dome to outdoor sconces and other practical applications.
23/4/2146 3D Modeling Using MVML Dome
Un-calibrated Un- known Un-synchronized
Conclusion
• Continuous depth map based MVS is prospective for its accuracy, robustness and flexibility
• The Vision Field and the MVML dome has still lots of interesting problems haven't been investigated
23/4/2147 3D Modeling Using MVML Dome
Acknowledgement
• Bennett Wilbrum
• Moshe ben-ezra
23/4/2148 3D Modeling Using MVML Dome
23/4/2149 3D Modeling Using MVML Dome
We hope for corporative researches on related topics
Welcome to visit BBNC and Tsinghua University!
Thank you!
宽带网数字媒体技术实验室Broadband Network & Digital Media Lab