Building Mobile AR Applications Using the Outdoor AR Library (Part 1)

Tutorial:Building Mobile AR Applications using

the Outdoor AR Library

Gun A. Lee Mark Billinghurst

The Human Interface Technology Laboratory New ZealandUniversity of Canterbury, New Zealand

21 Nov 2013 09:00-10:15

SYMPOSIUM ON MOBILE GRAPHICS AND INTERACTIVE APPLICATIONS

Welcome Mark Billinghurst

Director, HIT Lab NZ PhD, Univ. Washington AR, Interaction Design

Gun Lee Post Doc, HIT Lab NZ PhD, POSTECH AR, Android, Google Glass


Schedule

Introduction (Mark) Outdoor AR History (Mark) Design Guidelines (Mark) Example Applications (Mark) Building a Mobile Outdoor AR Application (Gun) Research Directions (Mark) Conclusions (Gun + Mark)

Introduction

Mark Billinghurst


What is Augmented Reality?

Definition [Azuma 97] Combines real and virtual images Is interactive in real-time Content registered in 3D

Azuma, R. T. (1997). A survey of augmented reality. Presence, 6(4), 355-385.


Outdoor AR Using mobile/wearable

systems to overlay AR content outdoors

Technical Requirements Tracking (GPS, compass,

etc) Display (handheld,

headmounted) Input devices Processing, networking


Applications

Tourism, Gaming, Architecture, Engineering, Etc

Outdoor AR History

Mark Billinghurst


Evolution of Mobile AR

Wearable AR

Handheld AR Displays

Camera phone

1995 1997 2001 2003 2004

Camera phone- Self contained AR

WearableComputers

PDAs-Thin client AR

PDAs-Self contained AR

Camera phone- Thin client AR


MIT Wearable Computing (1996)


Mobile AR: Touring Machine (1997)

University of Columbia Feiner, MacIntyre, Höllerer, Webster

Combines See through head mounted display GPS tracking Orientation sensor Backpack PC (custom) Tablet input

Feiner, S., MacIntyre, B., Höllerer, T., & Webster, A. (1997). A touring machine: Prototyping 3D mobile augmented reality systems for exploring the urban environment. Personal Technologies, 1(4), 208-217.


MARS View

Virtual tags overlaid on the real world “Information in place”


Backpack/Wearable AR

1997 Backpack AR Feiner’s Touring Machine AR Quake (Thomas) Tinmith (Piekarski) MCAR (Reitmayr) Bulky, HMD based


PCI 3D Graphics Board

Hard Drive

Serial

Ports

CPU

PC104 Sound Card

PC104 PCMCIA

GPSAntenna

RTK correction Antenna

HMDController

TrackerController

DC to DCConverter

Battery

WearableComputer

GPS RTK correction

Radio

Example self-built workingsolution with PCI-based 3D graphics

Columbia Touring Machine

Mobile AR - Hardware


HIT Lab NZ Wearable AR (2004) Highly accurate outdoor

AR tracking system GPS, Inertial, RTK system HMD

First prototype Laptop based Video see-through HMD 2-3 cm tracking accuracy


Image Registration

AR Stakeout Application


Wearable AR Video


Location Aware Phones (2008)

Nokia NavigatorMotorola Droid


Outdoor Information Overlay (2009)

Mobile phone based Tag real world locations

GPS + Compass input Overlay graphics data on live video

Applications Travel guide, Advertising, etc

Wikitude, Layar, Junaio, etc.. Android based, Public API released


Google Glass (2013)



View Through Google Glass

Always available peripheral information displayCombining computing, communications and content capture


Hardware CPU TI OMAP 4430 – 1.2 Ghz 16 GB SanDisk Flash,1 GB Ram 570mAh Battery

Input 5 mp camera, 720p recording,

microphone GPS, InvenSense MPU-9150 inertial

sensor Output

Bone conducting speaker 640x360 micro-projector display


Competitors Vuzix M100

$999, profession Recon Jet

$600, more sensors, sports Opinvent

500 Euro, multi-view mode Motorola Golden-i

Rugged, remote assistance

Design Guidelines

Mark Billinghurst


AR UI Design Consider your user Follow good HCI principles Adapt HCI guidelines for AR Design to device constraints Design for perception/info presentation Design for evaluation


Consider Your User Consider context of user

Physical, social, emotional, cognitive, etc Outdoor AR User

Probably Mobile One hand interaction Short application use Need to be able to multitask Use in outdoor environment Enhance interaction with real world


Adapting Existing Guidelines Mobile Phone AR

Phone HCI Guidelines Mobile HCI Guidelines

HMD Based AR 3D User Interface Guidelines VR Interface Guidelines


Clean Large Video View Large Icons Text Overlay Feedback

Applying Principles to Mobile AR


AR vs. Non AR Design

Design Guidelines Design for 3D graphics + Interaction Consider elements of physical world Support implicit interaction

Characteristics Non-AR Interfaces AR Interfaces

Object Graphics Mainly 2D Mainly 3D

Object Types Mainly virtual ob-jects

Both virtual and physical ob-jects

Object behaviors Mainly passive ob-jects

Both passive and active ob-jects

Communication Mainly simple Mainly complex

HCI methods Mainly explicit Both explicit and implicit


Maps vs. Junaio

Google Maps 2D, mouse driven, text/image heavy, exocentric

Junaio 3D, location driven, simple graphics, egocentric


Design to Device Constraints Understand the platforms and design for limitations

Hardware, software platforms Eg Outdoor AR game with handheld

Use large screen icons Consider screen reflectivity Support one-hand interaction Consider the viewing angle Do not tire users out physically Don’t require accurate tracking


AR as Perception Problem Goal of AR to fool human senses –

create illusion that real and virtual are merged

Depth Size Occlusion Shadows Relative motion Etc..


Which Object is Closest?


Depth CuesPictorial: visual cues

• Occlusion, texture, relative brightnessKinetic: motion cues

• Relative motion parallax, motion perspectivePhysiological: motion cues

• Convergence, accommodationBinocular disparity

• Two different eye images


Use the Following Depth Cues

Movement parallax Icon/Object size (for close

objects) Linear perspective

To add side perspective bar. Overlapping

Works if the objects are big enough

Shades and shadows Depends on the available

computation


Provide Perspective Cue

Eg ground plane grid



Depth Perception


Information Presentation

• Amount of information• Clutter, complexity

• Representation of information• Navigation cues, POI representation

• Placement of information• Head, body, world stabilized

• View combination• Multiple views


Twitter 360

www.twitter-360.com iPhone application See geo-located tweets in real world Twitter.com supports geo tagging


Wikitude – www.mobilizy.com

Blah

Blah

BlahBlah

Blah

BlahBlah

Blah

BlahBlah

Blah

BlahBlah

Blah

BlahBlah

Blah

BlahBlah

Blah

BlahBlah


Information Filtering


Information Filtering


Outdoor AR: Limited FOV


Possible solutions Overview + Detail

spatial separation; two views

Focus + Context merges both views into one view

Zooming temporal separation


Zooming panorama, Zooming Map

Zooming Views

Mulloni, A., Dünser, A., & Schmalstieg, D. (2010, September). Zooming interfaces for augmented reality browsers. In Proceedings of the 12th international conference on Human computer interaction with mobile devices and services (pp. 161-170). ACM.

Example Applications

Mark Billinghurst


HIT Lab NZ Building Viewer Architectural Application Loads 3D models

a OBJ/MTL format Positions content in space

GPS, compass Intuitive user interface

toolkit to modify the model Targeting museum guide/outdoor site

applications


2011 Christchurch Earthquake


Christchurch Today


CityViewAR

Using AR to visualize Christchurch buildings 3D buildings, 2D images, text, panoramas AR View, Map view, List view Available on Android market, iOS App Store



User Experience

While walking in the real world people can see text, 2D images and 3D content


List View

List of all assets


Map View

Icons for buildings, viewpoints, panoramas


Building History Data


Photographic Images


Panorama Images

360 degree photo bubbles


Augmented Reality View


Building a Mobile Outdoor AR Application

Gun Lee

Research Directions

Mark Billinghurst


Looking to the Future

What’s Next?


What’s Next Key Research Problems

Wide area tracking Input Methods/Displays Collaboration

Emerging Trends AR + Social Networking AR Standards/Platforms AR + Human Computing Scaling up


Robust Outdoor Tracking

Hybrid Tracking Computer Vision, GPS, inertial

Going Out Reitmayer & Drummond (Univ.

Cambridge)


Handheld Display

Reitmayr, G., & Drummond, T. W. (2006, October). Going out: robust model-based tracking for outdoor augmented reality. In Mixed and Augmented Reality, 2006. ISMAR 2006. IEEE/ACM International Symposium on (pp. 109-118). IEEE.


Meta Gesture Interaction

Depth sensor + Stereo see-through


Meta Video


Contact Lens Display Babak Parviz

University Washington MEMS components

Transparent elements Micro-sensors

Challenges Miniaturization Assembly Eye-safe


Contact Lens Prototype


Ego-Vision Collaboration

Google Glass camera + processing + display +

connectivity


Ego-Vision Research System

How do you capture the user's environment?

How do you provide good quality of service?

Interface What visual and audio cues provide best

experience? How do you interact with the remote user?

Evaluation How do you measure the quality of

collaboration?


Massive Multiuser Handheld/Outdoor AR for the first time

allows extremely high numbers of AR users

Requires New types of applications/games New infrastructure (server/client/peer-to-peer) Content distribution…


PERSONAL VIEW


Augmented Reality 2.0 Infrastructure


Leveraging Web 2.0 Content retrieval using HTTP XML encoded meta information

KML placemarks + extensions Queries

Based on location (from GPS, image recognition) Based on situation (barcode markers)

Syndication Community servers for end-user content Tagging

AR client subscribes to data feeds


AR Standards + Markup Languages

KHARMA and Argon ARML AREL Patterns if Interest X3D+ KML vs. RDF + Multimedia Markup

Languages



AR + Human Computation Human Computation

Real people solving problems difficult for computers

Web-based, non real time Little work on AR + HC

AR attributes Shared point of view Real world overlay Location sensing

What does this say?


Human Computation Architecture

Add AR front end to typical HC platform


Scaling Up

Seeing actions of millions of users in the world

Augmentation on city/country level


AR + Smart Sensors + Social Networks

Track population at city scale (mobile networks)

Match population data to external sensor data medical, environmental, etc

Mine data to improve social services


Scaling Up

AR on a City Scale Using mobile phone as ubiquitous sensor MIT Senseable City Lab

http://senseable.mit.edu/


WikiCity Rome

http://senseable.mit.edu/wikicity/rome/


Orange Data for Development

Orange made available 2.5 billion phone records 5 months calls from Ivory Coast

> 80 sample projects using data eg: Monitoring human mobility for disease

modeling

Conclusions

Mark Billinghurst, Gun Lee


Conclusions Outdoor AR hardware available

Handhelds (GPS, compass, camera), Wearables Many possible applications HIT Lab NZ Outdoor AR platform

Easy for building AR applications Multi-view support, Client/Server interface Cross platform (handheld, Glass, etc)


Next steps More tutorials on our website

http://www.hitlabnz.org/mobileAR Tags, LocationEvents, etc.

Projects on Google Glass http://arforglass.org One week workshop in Feb 2014

Stay tuned for server components Web-based Authoring Tool closed beta (Jan 2014)


http://www.hitlabnz.org/mobileAR


http://www.arforglass.org/


More Information Website

http://www.hitlabnz.org/mobileAR http://arforglass.org

Gun Lee [email protected]

Mark Billinghurst [email protected]