Hicss17 asakawa

Cognitive Assistant for The Real-world Accessibility

Chieko Asakawa IBM Fellow, IBM ResearchIBM Distinguished Service Professor, Carnegie Mellon University

© 2011 IBM Corporation2

Cognitive Ability of Visually Impaired People

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

0.8

1.0

1.2

1.4

1.6

1.8

2.0

2.2

2.4

2.6

2.8

3.0

3.2

3.4

3.6

音声速度(倍速)

再生

率 (%)

A

B

C

D

E

F

G

Reca

ll Rat

e (%

)

Speech speed

X X X X X X X X X X X X X XX

3 ©2016IBMCorporation


Information accessibility Mobility

Braille Digitalization

Paper Braille TypewriterComparison:

Braille and Printed Japanese Dictionary

Digital Braille System

Home Page Reader

6 © 2016 IBM Corporation

nThe first practical voice browsers in the world

nBecame a product in 1997 and translated into 11 languages

IBM Home Page Reader (1999) Japanese, Italian, French, German, Spanish, English (U.S.,

U.K.)

“Home Page Reader is my small window to the world. I can read, write, and access information.I can do everything to participate in society.”

— comment from a user, 1997

©2016IBMCorporation

Smart phone demo


Exponential Increase of Accessible Information (Conceptual Chart)


Paper BrailleDigital Braille

Voice Web access

1980 20001990 2010

Amou

nt o

f ac

cess

ible

info

rmat

ion

9

https://www.youtube.com/watch?v=cTHa_5Y5IA4

10

https://theses.lib.vt.edu/theses/available/etd-05072012-161514/unrestricted/DAngio_PC_D_2012_1.pdf

http://www.ted.com/talks/dennis_hong_making_a_car_for_blind_drivers

Blind Driver Challenge at the Rolex 24 At Daytona with Audio Description

Uber Self-driving Car Pilot


Walk by myself Finding discount signsFinding restaurantsShopping

The Second Challenge: Real World Accessibility

©2016IBMCorporation12

Cognitive Assistant

13

Augment missing or weakened abilities by the power of cognitive computing.


“甦るヒーローライブラリー第3集光速エスパー”, TCエンタテインメントより

Chika



https://www.ted.com/talks/chieko_asakawa_how_new_technology_helps_blind_people_explore_the_world

Foundation of Cognitive Assistant

©2016IBMCorporation16

LocalizationRadio-wave

VisionDead reckoning

RecognitionObjectsPeople

Environment

KnowledgePersonal

Social mediaEncyclopedic

InteractionSpeechGesture

Sonification

Localization

Quasi-zenith satellitesfor Japan

2018

http://qzss.go.jp/technical/technology/tech01_orbit.html

Accurate indoor localizationBLE Beacons + Dead reckoning

WIFI

Recognition of Environment

Vision-based EnvironmentRecognition

Infrared Depth Sensor on smartphonesGoogle Tango

https://youtu.be/Qe10ExwzCqk

Recognition of Things

Watson Vision API

coffeeespresso

wineglass

glasses potterydish

Oil filterChicken soup

Pen sharpener Spice bottle

Cristmas socks Cheetos

Beer bottle Diet coke

Photos for learning taken by blind subjects

Incorrect resultsBefore leaning

Correct resultsAfter learning

Personal Object Recognition

Honorable mention at CHI2017


Knowledge – Importance of Open data

Detailed pedestrian maps

Crowd prediction

RestaurantShops

Items

Temporal changes

21

Accessibility information

Interaction


Approaching to human accuracy

http://blogs.microsoft.com/next/2016/10/18/historic-achievement-microsoft-researchers-reach-human-parity-conversational-speech-recognition/#sm.0019gj85a12umd89ys32kew39p8d6

Autonomous Bus with Voice Dialog

https://www-03.ibm.com/press/jp/ja/pressrelease/50250.wss

デモ


Cognitive Assisting Robot

Manuela Veloso is leading the CoBot project. On November 18, 2014, the CoBot robots jointly reached 1,000km of autonomous localization and navigation.


Sep 2016 - August 2019 Awarded Amount: $1millionInvestigator:• Kris Kitani, Computer Vision, Robotics Institute, CMU• Manuela Veloso, Robotics, Robotics Institute, CMUAdvisor:• Chieko Asakawa, IBM Fellow, Robotics Institute, CMU

National Robotics InitiativeA Cognitive Navigation Assistant for the Blind

25

Implementation of a navigation assistant that uses a collection of sensing modalities and algorithms to guide a blind person through the knowledge landscape of an unfamiliar environment.

World Robot Summit


Industry robots Services robots Disaster response robots

26

Japanese government is planning a robot competition event in 2020, the same year, Olympics and Paralympics will be held in Tokyo.

http://www.nedo.go.jp/robot/about.html http://www.miraikan.jst.go.jp/exhibition/future/robot/robotworld.html#slide1

https://www-robotics.jpl.nasa.gov/tasks/taskImage.cfm?TaskID=236&tdaID=700043&Image=501

18th century, the mathematician Euler authored a leading textbook on mathematics after he became blind.

Captions for TV programs were invented for deaf people, but are now widely used in sports bars and to help immigrants, etc.

Based on Self-standing wheelchair.

2011, voice dialogtechnologies for smartphones

TelephoneKeyboard

Voice recognition

Voice dialogTV Captions

Voice synthesis and OCR

Modern Mathematics

Autonomic vehicles

Segway

“Blind driving challenges”

Early use of keyboards was a to help people with hand-and-motor disabilities in writing

First consumer product was a reading assistant for the blind in the 1970s.

1876, Alexander Graham Bell invented the phone after teaching hearing impaired people.

Innovations Flourish From Accessibility NeedsSmart

Machines

27

17th century, Dom Perignon. Champagne


1960sTalking Typewriter

1960s Talking Typewriter

1975 1403 Braille Printer

1984 Talking 3270 Terminal

1990 VoiceType™

2008 Social Accessibility

1998 ViaVoice®

2000 Accessibility Center

1997 Home Page Reader

1988 ScreenReader/DOS

2004 aDesigner

2007 Eclipse Accessibility Tools Framework

2009 Rich Internet Accessibility Technologies

1914 First IBMer with disabilities

IBM History of Accessibility


2012 Brazil Accessible Vocational Training Project

2011 Senior Workforce Project 1984Talking 3270 Terminal


2010 20302020

Amou

nt o

f Ac

cess

ible

real

wor

ld in

form

atio

n(c

once

ptua

l num

ber)

Future: Real-world Accessibility – Exponential Growth

Cognitive Assistant Lab @ CMU

Smartphone accessibility

Standardization?Core technology development

Mobile Cognitive ?

Apple Siri

Microsoft Seeing AI project

Google Tango Phone

Tokyo NihonbashiDemo

Watson Vision API

Adoption?Early days

2016Dec.

Tokyo OlympicsParalympics

Technology

Hicss17 asakawa