Dual PhotographyPradeep Sen | Billy Chen | Gaurav Garg | Stephen Marschner | Mark

Horowitz | Marc Levoy | Hendrik Lensch

Prashanth PM

Introduction

Dual Photography uses the concept of Helmholtz Reciprocity to interchange the lights and cameras in a scene.

Helmholtz Reciprocity

Bidirectional Reflectance Distribution Function (BRDF)

fr(ωi, ωo) (outgoing radiance / Incoming radiance)

fr(ωi, ωo) = fr(ωo ,ωi)

Helmholtz Reciprocity

First Relighting Work to make use of Helmholtz Reciprocity.

Radiance transfer between incoming and outgoing directions is

symmetric

So, we can generate an image from the point of view of the projector.

Primal Configuration c’ : Captured Image (nm x 1)

p’ : Projected Pattern (pq x 1)

T : Transport Matrix (mn x pq)

Dual Configuration c”: Virtual Projected Pattern (nm x 1)

p”: Virtual Captured Image (pq x 1)

TT: Transposed Transport Matrix (pq x nm)

Transport Matrix

T’’ = TT due to Helmholtz Reciprocity

Dual Photography is the act of multiplying the transposed matrix by a

desired lighting image vector.

Photography without a camera.

Image at right acquire using a 3x3 pixel

square scan across the projector.

Two photo resistors are used instead of

a camera.

Comparison with Debevec et. al. [2000]

In Debevec’s “Acquiring the Reflectance Field of a Human Face“, distant point light sources are assumed, and there is no spatial variation within the light source.

Hence, sharp shadows cannot be cast onto the scene.

Besides, a fewer number of light sources are required.

Optimizations in Capturing the T matrix.

Brute Force Method

Scan a single pixel per captured camera frame

Projector and Camera each have O(n6) pixels

Full T matrix would have O(n12) elements

HDR imagery required for scenes containing both specular and diffuse inter-reflections.

Even at a rate of 25 HDR images per minute, the capture process could take weeks !

Fixed Pattern Scanning

Assume each projector pixel affects a small, localized region of the camera. Divide the region into blocks.

Repeat exposures and encode each block’s illuminated pixels with a unique binary encoding.

Fixed Pattern Scanning

1 17

….. …..

T

1 17

Projector Pixels

1

17

We can determine the spot being lit using a truth table (bit encoding) for 3 locations.

4 2 1 Spot

0 0 0 0

0 0 1 1

0 1 0 2

0 1 1 3

1 0 0 4

1 0 1 5

1 1 0 6

1 1 1 7

Hamming error correcting codes are added to account for bit errors due to noisy measurement.

Limitations of Fixed Pattern Scanning

Requires one-to-one correspondence between camera and projector pixels.

This only supports direct illumination properly.

Diffuse Illumination can map many projector pixels to the same camera pixel.

This violates the initial assumption.

Adaptive Multiplexed Illumination

LEVEL 1

An 8x8 pixel project is used.

All Pixels are Illuminated.

Adaptive Multiplexed Illumination

LEVEL 2

The single block from Level 1 is subdivided

into four blocks.

Conflict is detected between regions 2 and 4

Additional subdivision is required

Adaptive Multiplexed Illumination

Conflicting blocks from Level 2 are not

co-scheduled in Level 3.

Some blocks can be scheduled in parallel.

Two new conflicts detected (6 & 12, 8 & 10).

The lower-leftmost block causes no illumination

and is culled

Adaptive Multiplexed Illumination

Final subdivision is applied in Level 4.

Many blocks are acquired in parallel.

Drawbacks of Adaptive Multiplexed Illumination

Does not do well for a scene with several diffuse inter-reflections.

A single projector pixel can affect the entire scene.

Hierarchical Assembly of the Transport Matrix

Dual from Adaptive Multiplexing Dual from Hierarchical AssemblyInflections between red wall and boxis almost lost.

Results

Primal Dual

Scene Relighting

Primal and Dual Images can be relit by multiplying T or TT by the desired matte as p’ or c.”

Scene Relighting

Primal Dual Dual with relit pattern

Dual – hand Dual – moving light beam

Scene Relighting

Projectors are not parallelizable, only one projector can be used at a time. But cameras can be parallelized to capture a 6D reflectance field.

Scene Relighting

Results

Dual – left lit Dual – right lit

Results

Dual – soft shadows Dual – hi res matte

Results

Dual – Soft Shadows Dual – Shadows cast by synthetic model

Limitations

Scenes containing significant global illumination

effects reduce parallelism of the adaptive method.

Limitations

Cameras have greater depth-of-field, better focus control, and more imaging controls

Limited depth-of-field can result in out-of-focus dual images

Limitations

Suppose the camera and projector are at a large angle apart from each other.

There may be many regions in the scene with no direct light transport between the camera and projector.

How to read your opponent’s card ?

Thank You

Questions ?