Kevin A. Shaw, Ph.D.

Chief Technology Officer

March 30th, 2014 Santa Clara, California, USA

2014 Embedded Vision Member Meeting kshaw@sensorplatforms.com

R2

• Consumer games – More stable attitude

• Augmented Reality (AR) – Needs improvement, better accuracy

• Indoor Navigation – Need better accuracy; lower power

• Hyper photography – Super resolution; intraframe deblur

• Robotics – Visual odometry to detect egomotion

– Always need better accuracy

6/2/2014 2

• Construction equipment – Perimeter safety

• Context awareness – Understanding users better

• Change from mobile to wearables – Shift from mostly-pocket to always-visual

– Digital eyewear makes a big difference

• Natural interfaces – Using the same wealth of information as

humans do to understand the world

6/2/2014 3

• Change to always-on consumer vision products

6/2/2014 4

• How to solve some of limitations of vision systems using some of these sensors

• Some limitations: – Lack of metric scale

• The Dollhouse problem

– Pose stability

• Feature point robustness

– Power consumption

• Suitable for mobile products?

6/2/2014 5

6/2/2014 6

Accelerometer

Gyroscope

Magnetometer

Barometer

Proximity

Amb. Light sensor

GPS

WiFi

Bluetooth

GSM/CDMA Cell

NFC

Camera (front)

Touch screen Camera (back)

20 sensors!

Humidity Colorimeter

CO2/VOC gas Microphones x 3

Fingerprint Thermal ambient

Sensor Fusion

7

• What are they? – MEMS are tiny silicon structures

– MicroElectroMechanical Systems

– Leveraging semiconductor toolsets Bosch

6/2/2014 8

• Measures dynamic acceleration – Very versatile.

– Result: Vibration, tilt, & position

– Low power 1-10uA

6/2/2014 9

Hydrogen atom = 1Å

MEMS displacement Resolution ~ 0.1Å

• Used to measure rotation

• Absolute orientation reference for gyroscope

• Power is moderate: 300-1500uA

6/2/2014 10

• Gyros don’t measure angle! – They measure the rate of change

– Body rates: rotation about each axis

• Rates are relative to starting point – Depend on Accel/Mag for start

• Integrate to get angle

• Power is high: 1-5mA or more

Gyro SEM 𝜃= 𝜔 𝑡 𝑑𝑡 + 𝜃0

6/2/2014 11

• Measures air pressure

• Air pressure indicates altitude

• Not good for absolute

• Resolution of 1-2 feet

• Low power: 1-5uA

Melexis.com

6/2/2014 12

Sensor Fusion

13

• We want to know Position and Attitude (pose). – Inertial and Vision systems can each help find this

6/2/2014 14

𝑃𝑜𝑠𝑒 = 𝑝 , 𝑞 = 𝑥, 𝑦, 𝑧, 𝑞0, 𝑞1, 𝑞2, 𝑞3,

• Position seems easy: double integrate

• Angle is only a single integration.

• No problem!

6/2/2014 15

𝑝𝑜𝑠𝑖𝑡𝑖𝑜𝑛 = 𝑣 𝑡 𝑑𝑡 + 𝑝0

𝑣𝑒𝑙𝑜𝑐𝑖𝑡𝑦 = 𝑎 𝑡 𝑑𝑡 + 𝑣0

𝜃= 𝜔 𝑡 𝑑𝑡 + 𝜃0

• Noise Random walk – Integrating noise causes a linear walk.

6/2/2014 Sensor Platforms Proprietary and Confidential Information 16

Measured Acc error [m/s/s] for an accelerometer when sitting still.

• Noise Random walk – Integrating noise causes a linear walk.

6/2/2014 Sensor Platforms Proprietary and Confidential Information 17

dttatv )()(

Measured)( ta

6/2/2014 Sensor Platforms Proprietary and Confidential Information 18

dttvtp )()(

dttatv )()(

Measured)( ta

• Gravity gets in the way

• And its big.

• Need to subtract it off, but….

6/2/2014 19

• Dead Reckoning – Over the past few years

significant progress has been made

– Stable solutions with consumer grade sensors

– Graph (right) uses stock sensors on Galaxy S3

– Pedestrian walking constraints aid solution

6/2/2014 20

Waypoints Measured Path

• Visual odometry – Visually tracking position (camera pose) through a space

• Tracking feature points is a powerful way to understand the world

6/2/2014 21

• Limitations – Can't tell size of objects: i.e. scale

• Doll house problem

– Hard to map points between frames over time

– Need cohesion over long time scales

– Need robustness in dark spaces & low-texture surfaces

– Need maintain vision lock (can get lost due to motion-blur)

– Enormous computational load (ready for mobile?!?)

• Can we aid the solution with more sensors?

6/2/2014 22

• Attitude estimates allow anticipated search space – Reduce computation for FP correspondence

• Power reduction with reduced/opportunistic frame rates – Can trust INS when not moving

– Or when spatial diversity is low

• Vision System can be turned on only when high resolution navigation/alignment is needed – PDR to Statue, VS for precise AR overlay

6/2/2014 23

• Need to find metric scale – Without it the world makes no sense

• Monocular / Binocular issues

6/2/2014 24

If you were a vision system, which one is real?

• Need some way to get extra scale information

– Binocular cameras (like humans do; monocular is cheaper)

– Reference object (hard to keep in sight; i.e. Ikea catalog)

– Location estimates (GPS is not available indoors)

– Mapped landmark (best but hard; humans do this)

– Inertial estimates (tend to drift, but commonly available)

– Depth cameras

6/2/2014 25

• State estimation from visual & inertial sources – Combined measurements & physical models

6/2/2014 26

],,,,ˆ,[ gak bbvqpx

• State estimation from visual & inertial sources

• Kalman filter – Recursive linear quadratic estimator

– Combined measurements & physical models

6/2/2014 27

],,,,ˆ,[ gak bbvqpx

– Closely coupled KF – Solve it all at once

• Computationally expensive (Order n2 or n3)

– Loosely coupled KF

• Estimate visual delta-pose

• Estimate inertial delta-pose

• Combine with KF

• But loose cross-correlations

6/2/2014 28

6/2/2014 29

[Weiss & Siegwart, 2007]

– Recursive vs Batch solution

• Kalman Filters are “recursive”; only one frame deep

– Batch solution:

• Compute solution across multiple frames

• Bundle Adjustment with well selected keyframes

• Much more stable, but computationally expensive

• Asynchronous to frame updates; non-uniform keyframes

6/2/2014 30

– Consumer sensors are cheap

• Free: they are already in place

• Contextual & Motional

– Need additional constraints

• Contextual constraints

– Are you moving?

» Easy for robots

» Harder for humans

» Not only for robots anymore

• Motion constraints

– Wheel constraints help (only on flat ground with no slippage)

– Pedestrian constraints (track steps, distance & direction)

6/2/2014 31

Demonstration 1: Ideal Vision & PDR Scenario

● Vision-only: 5%, PDR-only: 3%, Fused: 1.5%

● Use case: head-mounted AR, vision mapping

Initial scale estimate

Sensor Platforms, Inc. Confidential and Proprietary

32

End Start

Demonstration 2: Non-Ideal Vision/Ideal PDR

● Vision-only: 15%, PDR-only: 2%, Fused: 1%

● Use case: AR over large spaces

Vision Outage Initial scale estimate

Vision Outage

Sensor Platforms, Inc. Confidential and Proprietary

33

Start

End

Demonstration 3: Non-Ideal Vision/Non-Ideal PDR

● Vision-only: 8%, PDR-only: 15%, Fused: 5%

● Use case: intensive gaming

Initial scale estimate

PDR Outage

Sensor Platforms, Inc. Confidential and Proprietary

34

Start End

• Simultaneous Location & Mapping

6/2/2014 35

• Optical Image Stabilization (OIS) – Optical (in-lens) with inertial attitude tracking; gyro based

• Super resolution – Across multiple frames stabilized with pose tracking

• Deblurring – Within frame pose-detection and deconvolution

6/2/2014 36

– Across multiple frames stabilized with pose tracking

– Inertial data stabilizes the solution

6/2/2014 37

– Within frame pose-detection and deconvolution

6/2/2014 38