Tof Summary

對外秘

Time-of-Flight Depth Sensor based

3D Imaging for Future Display

2011.12.12Hyunjung Shim, Seungkyu Lee

Session I (8:30 am - 10:15 am)

Introduction to ToF Sensor Research

Principle of ToF Depth Sensor

Image Processing Algorithms for Depth Image Quality

Improvement

Session II (10:30 am - 12:15 pm)

Color/Depth Image based 3D Capturing and Modeling

Lighting and Reflectance Extraction from Color/Depth

Image: Inverse problem

Conclusion / Q & A

Course Schedule





Improvement





Conclusion / Q & A

Course Schedule

Depth Sensors

Sharp

Canesta

MESA

Fraunhofer

(&SIEMENS)

ITC-irst

EPFL

MIT

3DV Systems

NHK

Range: 15m

Resolution: 336×252

Accuracy: >23.5㎜@1m

Range : 3m

Resolution : 176×144

Accuracy : >30㎜

Range : 7.5m


Accuracy : 22㎜@3m

Range : 8m

Resolution : 64×4

Accuracy : >10㎜

Range : 9m

Resolution : 16×16

Accuracy : >90㎜

Range : 4m


Accuracy : 1.4㎜@1m

Range :12m

Resolution : 32×32

Accuracy : >30㎜

Range : 2.5m


Accuracy : >20㎜

Range : 10m


Accuracy : 25㎜@1m

External shutter

SPAD

Photogate

Photodiode

Depth Cameras

2001 2004 200720052002 2003 2006 2008

EPFL

MIT

MESA

3DV Sys.

CANESTA

Sharp

NHK

Fraunhofer

PMD Tech.

Z-CamTM Z-MiniTM DMC-100TM Z-SenseTM

SR2-A SR2-B SR-3000

DP200 DP300

Axi-Vision Camera HDTV Axi-Vision Camera

PMD[vision]® 19k160×120

3D CAM

Depth camera

PMD[vision]® 1k64×16

TOF Camera

Depth camera prototype

HDTV Axi-Vision Camera

ITC-irst

ITC-irst

AP

DP

hoto

gate

Photo

dio

de

Inte

nsifie

r

Depth camera prototype

SR-4000

ZCamTM

# of Publications From IEEEXplore

ToF Depth in Published Papers

0

5

10

15

20

25

30

2005 2006 2007 2008 2009 2010 2011

Calibration

Image Processing

3D Cap./Recon.

Interaction

Recognition,Detection(syst

em)

Overall

ToF-CV Workshop

(CVPR2008)

Special issue on Time-of-Flight Camera Based Computer Vision

(CVIU2010)

Human Activity Recognition from 3D Data Workshop

(CVPR2011)

IEEE Workshop on Consumer Depth Cameras for

Computer Vision (ICCV2011)

Applications on YouTube

Skeleton Detection– Body motion recognition & Interaction

– Pictorial body model fitting on the segmented foreground of depth image

– Real-time application for 3D game


3D Point Cloud– 3D Scene Reconstruction from 3D Point Cloud

– 2D pixel of depth image is a 3D geometry point seen from camera view

– 3D Imaging


3D Body Scanning & Virtual fashion using multiple Kinects

(www.tc2.com)– 3D reconstruction of real world objects

– Depth camera calibration & Point Clouds merging

– Space Interface for remote control by body gesture


Intel Labs (http://ils.intel-research.net/projects/rgbd)

Virtual World

3D Modeling

3D Capturing UX (1/3)

3D DSC/Camcorder 3D UCC on VW

Racing

Boxing

Embedded

3D Camera

80”

UD/3D Display

100Mbps

Network

Human Motion

Capture Sensor

VW


Action Based Game Virtual World on TV

Depth Camera

Video

Analysis

Video

Encoder

BroadbandNetwork

Event

Database

Video

Archive

Manager

Intelligent

Video

Security

Solution

Real-timeAlerts

Event Search/Event Statistics

Video

Analysis

Video

Encoder

BroadbandNetwork

Event

Database

Video

Archive

Manager

Intelligent

Video

Security

Solution

Real-timeAlerts

Real-timeAlerts

Event Search/Event StatisticsEvent Search/

Event Statistics

Real-timeAlert !!!

!!!

Motion Detection

Pedestrian

Detection

Alert

Depth Camera

Security Automotive






Improvement





Conclusion / Q & A

Course Schedule

3D Sensing Technologies

3D Sensing Technologies

Interferometry

Triangulation

Time-of-Flight

Depth (m)

Resolu

tion (

m)

10-6 10-3 100 103

10-9

10-6

10-3

100

Application Example

2D image

Depth map

3D model

Multi-view generation

New contents generation for 3D display

Time of Flight Sensing

Measure the round-trip time of emitted IR light

Object

R

Detector

IR Emitter

Contr

olle

r

Depth map

Distance Measurement

IR Modulation

irTOF

irTOFon

inTN

iTTnN

1

0

10

1

1

0 NN

NTT

T

TT

N

NonTOF

TOF

TOFon

TX0

TX1

Emitted

Light

Reflected

Light

Ton

TTOF

N0/n

N1/n

TX0

TX1

Emitted

Light

Reflected

Light

TTOF

N0/n

N1/n

2cos2sin

2cos2sin

2

1

00

TOFTOF

TOFTOF

TadaTaN

TadaTaN

10

1

1

0

2cos

2cos

NN

NTT

T

T

N

NonTOF

TOF

TOF

Pulse Light

Sinusoidal Light





Improvement

1. Intrinsic parameter acquisition and non-linear calibration

2. Range ambiguity

3. Depth noise modeling and denoising

4. Superresolution

5. ToF Motion Blur

Course Schedule

Demodulation-related Error

Caused by irregularity in the modulation process

use look-up table, B-Spline [1], and Polynomials [2]

Systematic Depth Errors

Figure 1. PMD, 1-4 m, B-Spline fitting [1]Figure 2. SR3100, 1-5m, IT 2ms - 32ms,

6 degrees polynomial fitting [2]

Err

or(

cm

)

Distance(m)

10

0

1 4

20

15

25

5

-5

-101.5 2 2.5 3 3.5 4.5

Demodulation-related Error

use 4 stage depth calculation [1]


Gate1

Gate2

Emitted NIR

Reflected NIR

Gate3

Gate4

A0

T0

t

d r·A0

Q1 Q1

Q2 Q2

Q3 Q3

Q4Q4

t

d

Y1≥0, Y2≥0 Y1<0, Y2≥0 Y1<0, Y2<0 Y1≥0, Y2<0

0 T0/2 T0 3T0/2 2T0

Y

─ Y1 = nQ1-nQ2

─ Y2 = nQ3-nQ4

1

2d

Y

Yarctant

Figure 3. Four Phase depth calculation Figure 4. Difference in depth calculation

Integration time-related Error

Due to the number of collected electrons during the

integration time worsen the repeatability problem


Figure 5. Colored 3D point cloud of a flat wall at a constant distance of 1 meter [2].

IT: 2ms IT: 4ms IT: 8ms

X(m) X(m) X(m)

Z(m

)

Z(m

)

Z(m

)

0.5-0.50.5-0.50.5-0.5

1

0.9

1.1

0.8

1

0.9

1.1

0.8

1

0.9

1.1

0.8

1.15

0.95

0.85

1.15

0.95

0.85

1.15

0.95

0.85

0 0 0

Pixel-related Error

Due to the different material properties of CMOS-gate and

amplitude-related error

use a Fixed Pattern Noise table


Figure 6. Fixed pattern noise, SR-2. IT

100ms, nominal distance 2.452 m [3]Figure 7. Depth-colored amplitude-related errors.

Depth image of a flat wall at 0.43 meters. Depth

overestimation can be observed due to low

illumination (borders of the image) [2].

0.41meter

0.43

0.45

Amplitude-related Error

Due to the non-uniformity of IR illumination and reflectivity

variation of objects

use a polynomial fitting model


Figure 8. Amplitude image of a planar

object with a ramp image. Parts of the

ramp are selected for calibration (blue

rectangle).

Figure 9. The depth samples (blue)

and the fitted model (green) to the

error

x(pixel)

y(p

ixe

l)

Amplitude

Err

or(

m)

0

0.001

0.003

0.004

0.002

0

10.5

Amplitude Correction

Light attenuates according to the law of inverse square


Figure 20. Distance-based intensity correction [18]

Temperature-related Error

Due to the response of the semiconductor to temperature

change

Wait for at least 4 min after turn on


Figure 10. Distance offset drift (fixed target) caused

by self-induced heating of the sensor. SR-2 [3]

Time(min)

Err

or(

m)

0.4

0.3

0 10 20 30 40 50 60 70

0.35

0.45

Light Scattering

Multiple light reflections between the lens and the sensor

Use scattering model [4] or anti-reflection material on lens

Non-systematic Depth Errors

Figure 11. Light scattering in TOF camera [4]

Figure 12. Scattering artifacts (a) Color image

(b) Background range image (c) Range image

with foreground. (d) Range image difference [4]

Multiple Light Reception

Due to interference of multiple light reflections

Use multipath interference model [5]


IR LED

Sensor Figure 14. Top view of the corner. The green points (a

laser scanner). The red points ( the ToF camera with

MPI, RMSE 57mm). The black points (compensated

by MPI, RMSE17mm ) [5]

Figure 13. Multipath Interference

Jump Edge Error

Due to multiple light reception

Use outlier rejection method [6][7]


Figure 15. Jump Edge Error

Motion Artifact (Blur)

Due to the movement during integration time

Detect pixels with phase deviation [14]


Figure 16. Motion blurring Figure 17. Motion artifact [14]

Consecutive positions Consecutive positions

Dynamic caseStatic case

Dis

tan

ce

(m)

Depth Folding (Phase Wrapping, Range Folding)

Due to the modular error in phase delay

Use multiple frequencies [17] or MRF with continuous

surface assumption [8]


Figure 20. Depth unfolding [8]

Emitted signal Incoming signal2

max2RR

)2sin( f ftA

f

R maxRR

)2sin( ft

Figure 18. Depth folding

Figure 19. Depth unfolding

using multiple frequencies [17]

No unified solution for handling all the errors!

Correct FPN and distance offset first

Utilize amplitude and color information (if exist)

Use constant integration time

Open research issues are here!

Depth range ambiguity, multiple light reception effect,

motion artifact in the complex scenes

Comments on Depth Measurement Errors

Bilateral Filter

uses weighted average of depth values from nearby pixels

Depth Noise Reduction

Figure 21. Bilateral filter kernel [9]

Figure 22. Bilateral filtering result

Non-Local Means Filter

uses weighted average of all pixels using the similarity of

squared neighborhoods


Figure 23. Non-local denoising result [7]

Learning-based Method

addresses depth discontinuity or low infrared reflectivity

uses Random Forest regressor trained with real-world

data [5]


Figure 24. Flying pixels due to

unsuitable reflectivity and large

depth discontinuities [6]

Figure 25. Artifact pixels [6]

Joint Bilateral Filter-based Method

refines depth values using color similarity [15]

Depth Super-resolution

(a) Color image 640x640 (b) Input depth map 64x64 (c) Refined depth maps 640x640 [15]

Markov Random Field-based Method

assumes that discontinuities in depth and color coexist


Figure 27. MRF-based depth super-resolution result [12]

Multiframe-based Method

models the image formation process from multiple depth

images


Figure 28. super-resolution result [11]

Filtering-based Method

smoothens the disparity map to hide occluded regions.

Novel View Synthesis

Figure 29. Depth map smoothing [17] Figure 30. Depth map smoothing [18]

Inpainting-based Method

fills the disoccluded region using image-inpainting

techniques.

Novel View Synthesis

Figure 27. Bi-layer inpainting [15]

Image warping Occlusion boundary

labeling

Foreground/background

segmentation

Exemplar-based inpainting

[19]

Region

to be inpainted

Foreground

Region

Background

Region

Segmented region Inpainted result Original imageWarped disparity

map

Labeled occlusion

boundary

ToF Motion Blur

Moving camera/object cause wrong depth calculation

Motion blur

Image sensor

Moving Object Moving Object

ToF Motion Blur

The characteristic of Tof motion blur is different from color

motion blur

Overshoot Blur

Undershoot Blur

Overshoot Blur

ToF Motion Blur

The characteristic of Tof motion blur is different from color

motion blur

Reflected IR

4-Phase signals inside ToF camera

TimeInteg.

1Q

2Q

3Q

4Q

Radiated IR

21

43arctannQnQ

nQnQtd

Depth calculation using the relation of 4-Phase signals

ToF Motion Blur

ToF motion blur model

21

43arctannQnQ

nQnQtd

))(())((arctan)(

2211

43

QmnmQQmnmQ

QnQnmtd

2

212211

43

)()(1

1)('

QQnQQQQm

QnQnmtd

)(

)(

2211

12

QQQQ

QQnm

ToF Motion Blur

)22(

)21(

)(

)(

11

1

2211

12

QQ

Qn

QQQQ

QQnm

(Normalized)

1Q

1Q̂

1)22(

)21(0

11

1

QQ

Q

ToF Motion Blur

How ToF motion blur can be detected / removed?

- Hardware modification

- Image processing methods using blur model

US7450220 Canesta, “Method and system to correct motion

blur and reduce signal transients in time-of-flight sensor

systems”

S. Hussmann et al. "Real-Time Motion Artifact Suppression in

TOF Camera Systems" IEEE Tran. on Instrumentation and

Measurement, 2011

O. Lottner et al. "Movement artefacts in range images of time-of-flight cameras" Int. Symposium Signals,

Circuits and Systems, 2007

M. Lindner and A. Kolb "Compensation of motion artifacts for time of flight cameras" Dynamic 3D Vision

Workshop, 2009

[1] New Insights into the Calibration of ToF-Sensors, CVPRW08

[2] Lock-in Time-of-Flight(ToF) Cameras A Survey, SJ11

[3] Calibration for Increased Accuracy of the Range Image Camera Swissranger, ISPRS06

[4] Real-time scattering compensation for time-of-flight camera, CVS07

[5] Multipath Interference Compensation in Time-of-Flight Camera Images, ICPR10

[6] Capturing Time-of-Flight Data with Confidence, CVPR11

[7] Robust Non-Local Denoising of Colored Depth Data, CVPRW08

[8] Range unfolding for time-of-flight depth cameras, ICIP10

[9] Bilateral Filtering for Gray and Color Images, ICCV98

[10] Spatial-Depth Super Resolution for Range Images, CVPR07

[11] High-quality Scanning using Time-Of-Flight Depth Superresolution, CVPRW08

[12] An Application of Markov Random Fields to Range Sensing, NIPS05

[13] Stereoscopic Image Generation Based on Depth Images for 3D TV, TBC05

[14] Movement Artefacts in Range Images of Time-of-Flight Cameras, ISSCS07

[15] Bi-layer inpainting for novel view synthesis, ICIP11

[16] Discontinuity-adaptive Depth Map Filtering for 3D View Generation

[17] A Time-Of-Flight Depth Sensor – System Description, Issues and Solutions, CVPRW04

[18] Measurements with ToF Cameras and Their Necessary Corrections, ISSCS07

[19] Region Filling and Object Removal by Exemplar-Based Image Inpainting, TIP04

[20] Extrinsic and Depth Calibration of ToF-cameras, CVPR08

References

Time-of-Flight Depth Sensor based

3D Imaging for Future Display

Session II

Geometry, Material, Lighting for Realistic Synthesis





Improvement





Conclusion / Q & A

Course Schedule

Camera Array based 3D Modeling (CMU)

- Synchronized 49 Cameras (9 on the ceiling + 10 on each wall)

- Video format: S-video, YCbCr 422, 640x480- One control PC + 17 digitizing PCs- Real time 3D Video/Audio Capturing- Arbitrary view synthesis

Input Color

3D Voxel Model

View Synthesis

a

b

c

a

b

c

Camera/Lens Array

-Angular/Spatial resolution trade-off

problem

-Camera size and usability problem

Camera Array, Stanford Univ. Plenoptic Camera, Adobe

Integral Photography

2D Projection image set capturing

(4D LF 2D image set)

3D Volume -> Surface

(6D(θ,Φ,x,y,z,λ) LF 4D(X,Y,x,y) LF)

2D Projection image set capturing

(4D LF 2D image set)

Lens Array Camera

Conventional Color Camera

Lens Array Camera

Multiview from Lens Array

A

B

C

213213

213

Multiview Display

Integral Photography Display

Color + Depth Camera Set (HHI)

- Heterogeneous Camera Set (Color + Depth Camera)

- Layered Depth Video (LDV) format creation

Bilateral Filtering

4Color + 1Depth Color Input

Depth Warping

Layered Depth Video

Occlusion Map

Depth based view synthesis

CGH from Multi-view / Color+Depth

NICT, Japan, 2008

Hokkaido University, Japan, 2007

Hokkaido University

NICT

3D Reconstruction from Color+Depth

- Color + Depth is one potential future 3D Camera

Multi-view Display

Integral Imaging Display

Holography Display

Examples

http://lightfield.stanford.edu/aperture.swf?lightfield=data/amethyst_lf/preview.zip&zoom=1

Color Discontinuity

Mixed Camera-Systems

ToF/RGB Geometric Alignment

Camera Calibration

Reprojection Results

Full Four-System Configuration

Multi-System Alignment

Segmented Figure

Rendered Figure & Background

Reprojected Figure-Mesh





Improvement





Conclusion / Q & A

Course Schedule

Considering the usability, we anticipate that the 3D sensing

architecture with scene analysis and synthesis algorithm will

be the most probable solution. Inverse rendering!

Realistic Rendering in Computer Graphics 3D Imaging for Future Display

Autostereoscopic DisplayStereoscopic Display

Stereo Camera

?Geometry,

Reflectance,

Lighting

Rendering

multiviews

Scene representation for realistic visualization

Realistic Rendering in Computer GraphicsInverse Problem

Photographs

Forward

Rendering

Inverse

Rendering

This slide is adapted from Koppal and Narasimhan

Geometry

Lighting Camera

Materials

Elements for realistic scene visualization


Geometry

Material

Lighting

Rendering

This slide is adapted from Xin Tong’s slide for material modeling,

originally from Ravi Ramamoothi.

70’s, 80’s: Splines

90’s: Range Data

10’s: ToF Depth Sensing



Geometry

Material

Lighting

Rendering

This slide is adapted from Xin Tong’s slide for material modeling.



Geometry

Material

Lighting

Rendering




Geometry

Material

Lighting

Rendering


Considering depth sensors for geometry modeling, we focus

on the inverse lighting and inverse reflectometry problems.


Geometry

Material

Lighting

Rendering


Inverse problem is very challenging. Materials exhibits complex characteristics, represented by high

dimensional functions.

It is hard to factorize the lighting and reflectance simultaneously.

( ill-posed )

Existing approach often assumes the surface

characteristics/illumination conditions to reduce the unknowns.

Realistic Rendering in Computer GraphicsChallenges

When dealing with real scenes, we need to account for

the noise of geometric model. Existing approach assumes the ideal/high resolution geometric

model while the true geometric model from real measurements do

suffer from noise and poor resolution.

Realistic Rendering in Computer GraphicsChallenges

Geometry

Material

Lighting

Rendering

Inverse ReflectometryMaterial Model

BSSRDF (8D)

BTF (6D)

BRDF (4D)

Homogeneous BSSRDF (5D)

Texture (2D)

Spatially VaryingBRDF (6D)

Lambertian (1D)

Q: Give the limited resource, how far can we achieve?

Sim

ple

Mate

rials

Com

ple

x

Mate

rials

Inverse Reflectometry# of Inputs

Few Many

Debevec et al. 01, 04

Ramamoorthi and Hanrahan 01Boivin and Gagalowicz 01

(# of Inputs)

Georghiades 03 Hara et al. 05 Nishino et al. 01, 05

Mahajan et al. 08 Yu et al. 97Zickler et al. 05

Gardner et al. 03 Mcmillan et al. 03

Munoz et al. 11

Weyrich et al. 09Dong et al. 10

Hullin et al. 10 Tong et al. 05

Sim

ple

Mate

rials

Com

ple

x

Mate

rials

Jensen 01

# of Inputs determines the possible applications Few inputs : Suitable for simple materials, dynamic scenes, consumer

electronics (CE) devices etc.

Many : Suitable for complex materials, static scenes, movie/scientific

applications etc.

Recent research trends move from many to few inputs,

from simple to complex materials.

Inverse ReflectometryMany Inputs

Capturing many photographs of real scenes (>1000) for

acquiring the reflectance of real scene Possible to model the real scene and to achieve the photorealism

Applied for movie special effects (e.g. Light Stage)

Impractical for general users (Not suitable for CE)

Inverse ReflectometryMany Inputs

Inverse ReflectometryFew Inputs

Photographs

Geometric model

Inverse

Rendering

Algorithm

Lighting

BRDF

Reflectance

Inputs

Approximating the material parameters from a sparse set

of photographs taken under uncontrolled environment Attractive to practitioners and general users

Quite challenging problems due to too many unknowns

Limited to homogeneous surfaces or simple materials

Inverse ReflectometryFew Inputs

Input Photographs

View Synthesis Illumination Synthesis

Light probe for inverse lighting Recovering high dynamic range imaging

Matte, mirror light probes

Assuming a class object (e.g. eye, face etc.), estimating

the lighting distributions

Inverse Lighting

# of Inputs

Few Many

Ramamoorthi and Hanrahan 01

Boivin and Gagalowicz 01

(# of Inputs)

Georghiades 03

Nishino and Nayar 05, 06

Mahajan et al. 08

Yu et al. 97

Zickler et al. 05

Spec

ific

Sce

ne

Gen

eral

Sce

ne

Debevec 98,01, 02

Wang and Samaras 02

Ramamoorthi et al. 01 Okabe et al. 04

Nishino et al. 01

Li et al. 09

Unger et al.03

Cohen et al. 01

Corsini et al. 08

Inverse LightingGeneral Scenes

Various light probes/light goniometrics are employed to

measure the lighting Impractical for general users (need to install the light probes while

capturing the scenes)

Shadows are another useful key for estimating the lighting

distribution Performance will depend on the shadow detection algorithm

Inverse LightingGeneral Scenes

A class-object model is assumed. Performance depends on the object detection/segmentation etc.

Lighting can be recovered up to the reflected light.

Inverse LightingSpecific Scenes

Inverse Problem In Practice

Inverse problem using a color/depth camera – A Color/Depth camera ready for the market

– Applying the inverse rendering algorithm onto such a device to reveal the

appearance of scenes

Few inputs– Compact parametric representation for materials

Complex materials and global illumination – Diffuse (Texture), glossy, specular (Direct illumination)

– Translucency, subsurface scattering (Indirect illumination)

Lack in details – Geometry enhancement ( motivated by super-resolution )

– Semantic approach

Noisy inputs– Accounting for noise models during inference

– Robust algorithm ( motivated by photometric stereo)


+






Complex materials and global illumination – Diffuse, glossy, specular (Direct illumination)



Inverse Reflectometry with Few Inputs[Homogeneous BRDF]

Given the geometry and few input images, solving the

inverse problem– Recovering both reflectance as well as lighting

– Assumptions: Known geometry, Distant illumination, Homogenous isotropic materials,

No shadows and interreflection

Ramamoorthi and Hanrahan [„01] employed a signal

processing framework to formulize the forward/inverse

rendering

Light ImageBRDF

Input Signal Output Signal

System


Reflection as convolution

2

2

id( )iL Lighting

( )oB

Reflected

Light Field

( , )i o

BRDF

( )iL ( , )i o id2

2

( , )oB

LL

i o

Global Coordinate = Local Coordinate Global Coordinate ≠Local Coordinate

'

o

'

i

'2/

2/

''' ,, ioiio dLB

'2/

2/

''' , ioii dL

Linear Shift Invariant System


Reflection as convolution

Analyzing well-posed conditions for inverse

lighting/reflectometry. High frequency in BRDF (Sharp highlights) Well-posed in inverse lighting

High frequency in Lighting distribution (Sharp features like point lights, edges, etc.)

Well posed in inverse reflectometry

2

2

id( )iL Lighting

( )oB

Reflected Light Field

( , )i o BRDF

( )iL ( , )i o id2

2

( , )oB

plllpl LB ,,

LB

LSI SYSTEM Convolution operator


Applying a nested algorithm for estimating unknown

lighting/BRDF

Photograph Rendered

Photograph Rendered

pll

pl

l

BL

,

,

ll

pl

plL

B

,

,

Inverse Reflectometry with Few Inputs[Homogeneous BSSRDF]

Estimating the BSSRDF from a single image [Munoz ‟11]

– Assumption: Optically thick materials, Dipole diffusion, A fixed a refraction index

– After all simplification, the unknown is the diffuse reflectance function Rd

ix

Aiiodotoo dAxExxRFxB ,,

j

bbjbdffjfd AErRAErR ,,

Dipole diffusion approximation

Sum of front & back irradiance maps


Estimating a diffuse reflectance function– Supposed that the lighting/shape is given, the front/back irradiance maps can be

computed

– Rd are expressed as a linear combination of basis functions, the piece-wise constant.

Solving a linear equation to compute coefficients cj

J

j jjd recrR1

k

bjkbfjkfij reKreKa ,,

J

j jjd rcrR1

e

bAx


Given the coefficients of diffuse reflectance function, it is

possible to rendering the target materials under varying

illumination/geometric conditions.

Inverse Lighting with Few Inputs[General scene]

Capturing an incident light field using a mirror-like light probe

0 stop -3.5 stop -7 stop

Mirror light probe

Generating HDR map for

incident lightingResponse function

[Debevec ‟97]

Multiple images with varying exposure time for HDR recovery

Capturing high frequencies in light distribution


Capturing an incident light field using a matte light probe[Wang and Samaras ‟02]

Input Detecting critical

boundariesReconstructed

A set of points

perpendicular to the light

direction

A single image with Lambertian object

Low frequencies in light distribution


Capturing an incident light field using a planar light probe[Alldrin and Kriegman ‟06]

A single image with a three-layered light probe

Low frequencies and some high frequencies in light distribution

Top Pattern: Sinusoidal printed on translucent sheet

Medium: Glass (0.096 in, Refractive index 1.52)

Bottom: Sinusoidal printed on Lambertian sheet


Capturing an incident light field from shadows [Sato et al. „02]

Input Regenerating shadows


Lighting distribution over the hemisphere

Estimating the light distribution

Inverse Lighting with Few Inputs[Specific scene]

Estimating an incident light field from a class-object – Faces, eyes, etc.

R=7.6mm

Anatomical model for human eye

Gaze detection

Inverse Lighting with Few Inputs[Specific scene]

Estimating an incident light field from a class-object – Faces, eyes, etc.

Extracting an environment map (incident light)

From many to few inputs– Compact parametric representation for materials

Complex materials and global illumination– Diffuse (Texture), glossy, specular (Direct illumination)






Problem Definition - Reminder

Inverse Problem with Complex Illumination

Estimating both the lighting and reflectance from a single

image

Lambertian surfaces to Non-Lambertian surfaces – Recent study extends the work to recover a semiparametric reflectance

model, more general than empirical models [Chandraker and Ramamoorthi „11]

– Necessary to recover the texture

Photograph Rendered

– Related work : Ramamoorthi and Hanrahan [„01]

– Existing work derived the theoretical analysis of

inverse problem, factorizing the reflectance and

lighting from a single image

Inverse Problem with Complex Illumination[General BRDF]

General BRDF for accounting a variety of materials

– Proposing a semiparametric reflectance model, more general than empirical

models, as a sum of univariate functions

– They proved that 2-lobe BRDF can be uniquely identified by a single input

image

– In various empirical model, α can correspond to light vector, half way vector

or view vector

Estimating the reflectance functions – Unknowns become αi and the form of function fi

– Solving a regression problem

K

i

T

ii nfn1

)()(

2

1,)()(min

K

i

T

iif

nfnii

[Chandraker and Ramamoorthi „11]

Inverse Problem with Complex Illumination[General BRDF]

Comparable to ground truth Better than the empirical

model

Input Relighting

Ground truth Error

Input Proposed

Ground truth T-S Model

Inverse Problem with Complex Illumination[Global Illumination]

Inverse light transport to extract the global illumination

effects (e.g. interreflection) – Exploiting the duality of forward/inverse light transport

[Carroll et al. „11]


Extracting an indirect illumination iteratively – Coefficients of S-1 is the result of Neumann series


Separating the indirect illumination out

Inverse Problem with Complex Illumination[Textured Surface]

Accounting for spatial variants in BRDF – Assuming Bivariate BRDF

– Input : About one hundred images under varying illumination conditions

[Alldrin et al. „08]


Complex materials and global illumination – Diffuse (Texture), glossy, specular (Direct illumination)







+

Inverse Problem with Noises

Depth/Color misalignment – To achieve the same vantage points for a depth image and a color

image

Depth noises upon material properties – Most depth sensors suffer from the depth distortion upon the

material characteristics

Missing data


Robust approach for photometric stereo – A class PS problem assumes the Lambertian materials, without

shadows

– In reality, the surface is mostly non-Lambertian and includes

shadows

– Robust approach handles the non-Lambertian illumination and

shadows as errors

Rank minimization problem– Convex Lambertian surfaces represented by at most 3 rank structure

– Formulating the problem by minimizing the rank with sparse error

constraint

– Solution via convex programming

[Wu et al., ACCV‟10]

EADtsEEA

..min1*,

A

EADtsErankEA

..min0,

A

[Candes, Li, Ma, and Wright‟09]


[Wu et al., ACCV‟10]

Conclusion

It is time to merge the advanced graphics and vision

research for 3D imaging technology.

Practical issues remain for general scenes, complex

materials, few input images. – Parametric vs. Data-driven representation for materials

– Complex illumination

– Physically accurate constraints for appearance

– Robust approach for geometry reconstruction

For outdoor scenery, we need to resolve the range limit on

depth camera – Interference on IR

– Range limit

Q&A

Thank you!

Documents

Tof Summary