Raskar stanfordextremecompuimagingapr2016

A Proposed Strategy

Camera CultureRamesh Raskar

Ramesh RaskarAsso. ProfessorMIT Media LabExtreme Computational Imaging

Mohit, Shah, Angelino, Sinha

Provide oral health solutions via a tooth brush that takes achieves x-ray like screening4

Vision is important for day labor for his employment5

Huang, Wetzstein, Barsky, Raskar. SIGGRAPH 2014.Eyeglasses-free Display: Towards Correcting Visual Aberrations with Computational Light Field Displays.

EyeSelfie, Swedish et al SIGGRAPH 2015

Siggraph 2015 Paper: EyeSelfie: Swedish, Roesch, , Raskar8

ReplacementExpansionPredictionPredictive Modeling of Future Health

Theia: API for Diabetic Retinopathy ScreeningOpen API and Documentationtheia.media.mit.eduDrag and Drop Web DemoSwedish et al

Visual Urban SensingNaik, Glaeser, Hidalgo, Raskar (email: [email protected])Crowdsourcing

Computer Vision+

3.6/10Sentiment Analysis for Street-level Images

Quantifying Urban ChangeApplications: Real estate, Data-driven urban policy and planning

DeclineImprovementstreetscore.media.mit.eduHow to use computer vision and crowdsourcing to evaluate the appearance of cities across the world at street-level resolution and global scale?

11

Smartphone spectroscopy for consumer applications

Rechargeable batteryArduino pro miniBluetooth moduleSpectrometer chipLEDFilter

Light switch

Power switch

Application in rapid testing of food quality: Chlorophyll fluorescence in fruitsAnshuman Das, Akshat Wahi, Ishan Kothari and Ramesh Raskar

Hand-held 3D Imager for biomedical applications

3D imaging of throat and tonsils for evaluation of obstructive sleep apneaAnshuman Das, Tulio Valdez, Ramesh Raskar et. Al., Biomed. Opt. Express7, 1445-1457 (2016)3D imaging of the tympanic membrane for accurately detecting ear infections

Anshuman Das, Julio Estrada, Ramesh Raskar et. Al., Proc. SPIE, 9303, 93031F (2015)

Health and Wellness Interfaces

Positions Open (tiny.cc/ccjobs)

Collaboration/Expt

Co-funding

16

http://cameraculture.infohttp://Fb.com/cameraculture MIT Media Lab

Raskar, Camera Culture, MIT Media Lab

Bit HackingPhoton HackingComputer VisionOpticsSensors

Signal ProcessingDisplaysMachine LearningComputational Light TransportComputational PhotographyIllumination

My work involves creative new ways to play with light by co-designing optical and digital processing.

My work lies at the INTERSECTION of processing of photons and processing of bits.

At MERL, I transformed the field of computational photography, with key papers and impact on productsAt Media Lab, I invented a new field computational light transport 18

DiffractionDynamic RangeDescattering

DefocusDepth

20

MilliMicroNanoPicoFemtoAttoConquer .. Time

If I turn this laser point ON and OFF in a trillionth of a second .. i.e. in a few femto-second ..

21

MilliMicroNanoPico

AttoFemto-graphyConquer .. Time NoiseSignal

I will create a packate of photons .. less than a milimeter long .. Bullet of light .. Traveling a million times faster than ord bullet22

Light in Slow Motion .. 10 Billion x Slow

Mitsubishi Electric Research LaboratoriesSpecial Effects in the Real WorldRaskar 2006

What if we send this bullet of photons to shatter into this bottleHow does light look in slow motionThe whole sequence is less than a nanosecond stretched to about 10 seconds i.e. 10 billion times slowed downIf an oridinary bullet, same distance at such a slow motion video, do you know how long you will have to sit here to watch the movie? A day, a week .. a whole year .. it will be a very boring movie ..

Clean_coke_mulSlow2v1.mov

23

Velten et al, Siggraph 2013


Andreas Velten, Di Wu, Christopher Barsi, Everett Lawson


25

Raw Data

so we have to use a stroboscopic sampling method to fire the bullet millions of times and do clever sync and record raw data, and use computational techniques to to weave together femto-video.

Worlds fastest videosSame photo about 10 million times to get enough photonsWorlds fastest slowest camera ..

26

Can you seearound corners ?

Superman can fly and heroes can becomes invisible .. But what about a new power for a future super-hero ..To see around corners .. 27

2nd BounceMulti-path Analysis1st Bounce3rd Bounce28

My idea is to use the multiple bounces of light i.e. echoes of light.

Echoes of sound -> Echoes of light

We all know about echoes of sound.But sounds travels slow and we can actually hear the echoesLight travels fast so we need specialized hardware to listen to these echoes.So we end up using light sources and cameras that run at a trillion frames per second (not a million and not a billion, but trillion)28

Collision avoidance, robot navigation, 29DARPA REVEAL Program by Dr.PredragMilojkovic

Motion beyond line of sightPandharkar, Velten, Bardagjy, Lawson, Bawendi, Raskar, CVPR 2011 30

, bronchoscopies, 31

And to see deep inside the body with endoscopes that look in narrow channels in lungs Or colons

Colonoscopy/cardioscopy32

Identify Materials

34

Naik, Zhao, Velten, Raskar, Bala, (SIGGRAPH Asia 2011)BRDF from a single viewpoint


Single Photon Sensitive Imaging

Gariepy et al. Nature Comm (2015)

Millisecond

Microsecond

NanoSecond

PicoSecond

FemtoSecond

AttoSecond

RFlowCAT-scanPhone3DReading BookFrequencyTime ResolutionLook Around CornersAudioRFXrayLightTHzFLIM

37

Multi-Dimensional Light Transport5-D Transport

38

Nanophotography70 picosecond resolution

[Kadambi et al 2013]

40

Real-time Localization

ImagingReal Time Localization

Kadambi, Zhao, Shi, Raskar. "Occluded Imaging with Time of Flight Sensors." ACM ToG 2016 (Pres. at SIGGRAPH)

Wavelength vsShininess

Macroscopic Interferometry

Optical Interferogram

Electronic Interferogram

Kadambi, Schiel, Raskar CVPR 2016 (Oral)

Macroscopic InterferometrySNR Benefits

Works at long-range

Obeys theoretic boundsfrom optical coherence

Macroscopic Interferometry

Depth Sensor

Kadambi, Taamazyan, Shi, Raskar, ICCV 2015Polarized 3D

MitigateExploitCreate

Scattered Light

48

FLIMSatat et al. Nature Comm 2015Bhandari et al. 2015

49

Application: Fluorescence Lifetime ImagingToF Imaging : Relative Phase Ecodes Distance InfoFLIM : Relative Phase Ecodes Lifetime Info

Probing FunctionResponse

Time Domain FLIM

ToF ImagingFLIM Imaging

Frequency Domain FLIM

ExcitationFluorescence

Lifetime

Parameterizes Lifetime

Application: Fluorescence Lifetime ImagingBhandari, Barsi, Raskar, 2016Blind and reference-free fluorescence lifetime estimation via consumer time-of-flight sensors

Location + Fluoro Lifetime Behind Diffuser

Satat et al. Nature Comm (2015)Bhandari et al. 2015

Imaging Behind Thick Scattering Media

1.5cm Thick Diffuser

Hidden SceneReconstruction with ToFSatat et al. Imaging Through Thick Medium Using Time-Resolved Measurement COSI (2015)

Mohit, Shah, Angelino, Sinha

Provide oral health solutions via a tooth brush that takes achieves x-ray like screening54

Optical BrushHeshmat, Lee, Raskar,,Nature Scientific Reports, 2016

55

Heshmat, Lee, Raskar,,Nature Scientific Reports, 2016Optical Brush

Sanchez, Heshmat, Reza, Romberg, Raskar 2015

Ancient text witout damaging books58

Sanchez, Heshmat, Reza, Romberg, Raskar 2015

Bedri, Gupta, Mohit et alTime-of-Flight Microwave CameraCharvat, Temme, Feigin, RaskarNature SciRep 2015

Kadambi, Bhandari et al

How can we drive on foggy, crowded streetsAs if it is a sunny day on a clear road63

Millisecond

Microsecond

NanoSecond

PicoSecond

FemtoSecond

AttoSecond

RFlowCAT-scanPhone3DReading BookFrequencyTime ResolutionLook Around CornersAudioRFXrayLightTHzFLIM

64

Multi-Dimensional Light Transport5-D Transport

My Photo

24 roll film for one year

65


Technology

Raskar stanfordextremecompuimagingapr2016