CS 591I / CS791B Computational Photography
CS 691B Computational PhotographyInstructor: Gianfranco DorettoModeling LightWhat is light?Electromagnetic radiation (EMR) moving along rays in spaceR() is EMR, measured in units of power (watts) is wavelengthLight:Travels farTravels fastTravels in straight linesInteracts with stuffBounces off thingsIs produced in natureHas lots of energy -- Steve Shafer
Figures Stephen E. Palmer, 2002What do we see?3D world2D image
What do we see?3D world2D imagePainted backdropOn Simulating the Visual ExperienceJust feed the eyes the right dataNo one will know the difference!Philosophy:Ancient question: Does the world really exist?Science fiction:Many, many, many books on the subject, e.g. slowglass from Light of Other DaysLatest take: The MatrixPhysics:Slowglass might be possible?Computer Science:Virtual Reality
To simulate we need to know:What does a person see?The Plenoptic FunctionQ: What is the set of all things that we can ever see?A: The Plenoptic Function (Adelson & Bergen)
Lets start with a stationary person and try to parameterize everything that he can see
Figure by Leonard McMillanGrayscale snapshotis intensity of light Seen from a single view pointAt a single timeAveraged over the wavelengths of the visible spectrum(can also do P(x,y), but spherical coordinate are nicer)
P(,)Color snapshotis intensity of light Seen from a single view pointAt a single timeAs a function of wavelength
P(,,)A movieis intensity of light Seen from a single view pointOver timeAs a function of wavelength
P(,,,t)Holographic movieis intensity of light Seen from ANY viewpointOver timeAs a function of wavelengthP(,,,t,VX,VY,VZ)
The Plenoptic FunctionCan reconstruct every possible view, at every moment, from every position, at every wavelengthContains every photograph, every movie, everything that anyone has ever seen! it completely captures our visual reality! Not bad for a functionP(,,,t,VX,VY,VZ)
Sampling Plenoptic Function (top view)Just lookup -- Quicktime VRRayLets not worry about time and color:
5D3D position2D directionP(,,VX,VY,VZ)
SurfaceCamera No Change in
RadianceLighting
How can we use this?14Ray ReuseInfinite lineAssume light is constant (vacuum)
4D2D direction2D positionnon-dispersive mediumOnly need plenoptic surface
vuSynthesizing novel viewsLumigraph / LightfieldOutside convex space
4DStuffEmptyLumigraph - Organization 2D position2D directionuLumigraph - Organization 2D position2D position
2 plane parameterizationusLumigraph - Organization 2D position2D position
2 plane parameterizationsuvu,vs,ttu,vs,tLumigraph - OrganizationHold u,v constantLet s,t varyAn image
u,vs,tLumigraph / Lightfield
Lumigraph - Capture Idea 1Move camera carefully over u,v planeGantrysee Lightfield paper
u,vs,tLumigraph - Capture Idea 2Move camera anywhereRebinningsee Lumigraph paper
u,vs,tLumigraph - RenderingFor each output pixeldetermine u,v,s,t
eitheruse closest discrete RGBinterpolate near valuesusLumigraph - RenderingNearestclosest uclosest sdraw it
Blend 16 nearestquadrilinear interpolationusStanford multi-camera array640 480 pixels 30 fps 128 cameras
synchronized timingcontinuous streamingflexible arrangement
28Light field photography using a handheld plenoptic camera
Ren Ng, Marc Levoy, Mathieu Brdif,Gene Duval, Mark Horowitz and Pat Hanrahan29Conventional versus light field camera
30Conventional versus light field camera
uv-planest-plane31Prototype camera4000 4000 pixels 292 292 lenses = 14 14 pixels per lens
Contax medium format cameraKodak 16-megapixel sensor
Adaptive Optics microlens array125 square-sided microlenses
32
33
Digitally stopping-downstopping down = summing only the central portion of each microlens34
Digital refocusingrefocusing = summing windows extracted from several microlenses
35Example of digital refocusing
36Digitally moving the observermoving the observer = moving the window we extract from the microlenses
37Example of moving the observer
38
Moving backward and forward
39
On sale now: lytro.com
http://www.lytro.com/science_inside
http://www.lytro.com/living-pictures/282
3D LumigraphOne row of s,t planei.e., hold t constants,tu,v3D LumigraphOne row of s,t planei.e., hold t constantthus s,u,va row of imagessu,vby David Dewey
P(x,t)2D: ImageWhat is an image?
All rays through a pointPanorama?ImageImage plane
2DpositionSpherical PanoramaAll light rays through a point form a ponoramaTotally captured in a 2D array -- P(q,f)Where is the geometry???
See also: 2003 New Years Evehttp://www.panoramas.dk/fullscreen3/f1.html Other ways to sample Plenoptic FunctionMoving in time: Spatio-temporal volume: P(q,f,t)Useful to study temporal changesLong an interest of artists:
Claude Monet, Haystacks studiesSpace-time images
Other ways to slice theplenoptic functionxytThe Theatre Workshop Metaphor
desired image(Adelson & Pentland,1996)PainterLighting DesignerSheet-metalworkerPainter (images)
Lighting Designer (environment maps)
Show Naimark SF MOMA videohttp://www.debevec.org/Naimark/naimark-displacements.movSheet-metal Worker (geometry)
Let surface normals do all the work! working togetherWant to minimize costEach one does whats easiest for himGeometry big thingsImages detailLighting illumination effects
clever Italians Slide CreditsThis set of sides also contains contributionskindly made available by the following authors Alexei Efros Richard Szelisky Michael Cohen Stephen E. Palmer Ren Ng
