Wiimote Infrared DetectionGreg Peaker

Max Chiswick

Minority Report Video

Project Goal and Setup• Use infrared for gesture recognition

– Works regardless of light and skin tone– Infrared detector: Wiimote– Infrared emitter: LED attached to hand

• Performs gesture recognition• Use gestures to control computer

applications

Wiimote Features

• Built in 1024x768 infrared camera– Processes images using system on

a chip and sends post-processed information to computer (or Wii)

• Connects to PC with Bluetooth• Wii Wiimote Usage:

– Wiimote moves around– Locates two IR LED emitter arrays

in sensor bar• Our Wiimote Usage:

– Wiimote stays still– IR emitter on finger moves around

Approach and Design

• Use off-the-shelf and inexpensive components– Wii remote– Infrared emitter: LED, resistor, battery power– Free, open source software

• Wiimote detects and tracks IR spots• Wiimote connects to computer through Bluetooth

– Sends data in ASCII at 100 Hz• Library parses ASCII to obtain easily readable parameters

– Not used: buttons pressed, acceleration, intensity– Used: (x,y) coordinates

• Code in Visual C#– Interpret user motions/Gesture Recognition– Control Windows Media Player and PowerPoint

How It Works• Software tracks (x,y) coordinates of IR led• Determine start time and coordinates• Create array of all coordinates in gesture

array from start to end– Sampled >100 times per second

• Determine end time and coordinates• Directional movements of finger control

windows applications

How It Works (2): Interpreting Start and Stop of Motion• Start gesture by

– Making infrared signal visible– Holding signal in place for about 2 seconds

• Program examines last 3 samples every second– If samples are within threshold, begin gesture– If samples are not (e.g. random movements), do not

begin gesture• Once gesture begins, program creates array that keeps

track of all (x,y) coordinates• End gesture by holding signal in place for about 2

seconds• Program continues to examine last 3 samples every

second– When it finds they are outside of threshold, continues

gesture– When it finds they are within threshold, ends gesture

How It Works (3): Determining Direction Used• Compare final (x,y) to initial (x,y)

• Obtain variables x_diff and y_diff• If x_diff > threshold, define as East or West• If y_diff > threshold, define as North or South• If neither > threshold, no motion• If one > threshold, basic North, South, East, or West

direction• If both > threshold, see if one direction is significantly

greater than other– Interpret this as only the larger one– Example: 100 units east and 12 units north probably

intended to be only east• If both > threshold and neither significantly greater than

other– Create line between start and end points of gesture array– Count every (x,y) element between start and end and

count them as above or below line– Use this info to determine, for example, whether gesture

is South-East or East-South

User Movements• 12 total possibilities

– Basic 4 directions– Direction combinations (order matters)

• North-East and East-North are different

Applications• Use directional movements for:

– PowerPoint• Next Page (North or East), Previous Page

(South or West)– Media Players

GestureSouth DownNorth UpWest LeftEast Right

GestureSouth – West PreviousNorth - West Volume DownNorth - East Volume UpSouth - East Next

GestureWest – South MuteWest - North MuteEast - North PauseEast - South Stop

Next Steps• Hardware

– Better define and increase range– Smaller IR transmitter/receiver assembly– Use reflection– Two finger or more finger tracking allowing

increased movements and novel applications

• Software– Support applications such as web browsers,

Google Earth, etc.– Use as mouse cursor with clicking ability– Better gesture recognition using machine

learning

IR Computer Vision TodayMicrosoft Surface

iBar

The computer's "vision" is created by a near-infrared, 850-nanometer-wavelength LED light source aimed at the surface. When an object touches the tabletop, the light is reflected to multiple infrared cameras with a net resolution of 1280 x 960, allowing it to sense, and react to items touching the tabletop.

iBar is an interactive, 10 meter long bar-counter using multi-touch technology. iBar has an intelligent tracking system to detect all objects touching the surface. Integrated video-projectors can project any content dynamically with the movements on the counter. The software is flexible and can be adjusted and enhanced. Objects can be illuminated at their position or virtual objects can be “touched” with your fingers. Every glass, cellphone, car key, cup, business card or even fingers are recognized when they touch the bar-counter.

Demo

GestureSouth DownNorth UpWest LeftEast Right

GestureSouth – West PreviousNorth - West Volume DownNorth - East Volume UpSouth - East Next

GestureWest – South MuteWest - North MuteEast - North PauseEast - South Stop