Download pdf - AAU SUMMER SCHOOL - Aalborg Universitetkom.aau.dk › ~zt › courses › SocialRobot_SummerSchool › Lecture10...5 AUGUST 2015 AALBORG UNIVERSITY 3 • Extended functionality, e.g

05/08/15

1

AAU SUMMER SCHOOL

PROGRAMMING SOCIAL ROBOTS FOR HUMAN INTERACTION

L E C T U R E 1 0 M U LT I M O D A L H U M A N - R O B O T I N T E R A C T I O N

1 . I n t r o d u c t i o n t o R o b o t O p e r a t i n g S y s t e m ( R O S ) 2 . I n t r o d u c t i o n t o i S o c i o B o t a n d N A O r o b o t , a n d d e m o s 3 . S o c i a l R o b o t s a n d A p p l i c a t i o n s 4 . M a c h i n e L e a r n i n g a n d P a t t e r n R e c o g n i t i o n 5 . S p e e c h P r o c e s s i n g I : A c q u i s i t i o n o f S p e e c h , F e a t u r e E x t r a c t i o n a n d S p e a k e r L o c a l i z a t i o n 6 . S p e e c h P r o c e s s i n g I I : S p e a k e r I d e n t i f i c a t i o n a n d S p e e c h R e c o g n i t i o n 7 . I m a g e P r o c e s s i n g I : I m a g e A c q u i s i t i o n , P r e - p r o c e s s i n g a n d F e a t u r e E x t r a c t i o n 8 . I m a g e P r o c e s s i n g I I : F a c e D e t e c t i o n a n d F a c e R e c o g n i t i o n 9 . U s e r M o d e l l i n g 1 0 . M u l t i m o d a l H u m a n - R o b o t I n t e r a c t i o n

COURSE OUTLINE

5 A U G U S T 2 0 1 5 A A L B O R G U N I V E R S I T Y 2

05/08/15

2

“ IN THE CONTEXT OF HUMAN–COMPUTER INTERACTION, A MODALITY IS THE CLASSIF ICATION OF A S INGLE INDEPENDENT CHANNEL OF SENSORY INPUT/OUTPUT BETWEEN A COMPUTER AND A HUMAN. A SYSTEM IS DESIGNATED UNIMODAL IF IT HAS

ONLY ONE MODALITY IMPLEMENTED, AND MULTIMODAL IF IT HAS MORE THAN ONE. ”

KARRAY, FAKHREDDINE, ET AL. "HUMAN-COMPUTER INTERACTION: OVERVIEW ON STATE OF THE ART." (2008).

MULTIMODAL INTERACTION – WHAT?


•  Extended func t iona l i t y, e .g . we can speak to the robo t ins tead o f t yp ing

•  Human-Human l i ke commun ica t ion

MULTIMODAL INTERACTION – WHY?


05/08/15

3

Robus tness aga ins t no ise •  Data f rom one moda l i t y m igh t be ve ry no isy, however the res t a re

no t – comb ine the moda l i t i es

•  Person Iden t i f i ca t ion : background mus ic i s co r rup t ing recorded speech , however v i s ion i s una l te red .

•  Speech Recogn i t ion : background mus ic i s co r rup t ing recorded speech , however v i s ion can be used to recogn ize l i p movements and c lass i f y words

•  How to know wh ich moda l i t y to “ t rus t ”?



•  Prov ide new in fo rmat ion , wh ich cou ld no t be p rov ided by ind iv idua l moda l i t i es

•  Combina t ion o f sound + fac ia l express ion = emot ion

•  Learn ing : •  Somet imes on ly one moda l i t y i s ava i lab le , bu t no isy

•  Use knowledge f rom one moda l i t y to re - t ra in /adap t mode l i n o ther doma in

•  Examples : Person Iden t i f i ca t ion , D i rec t ion o f A t ten t ion



05/08/15

4

•  iSoc ioBot

•  Research p ro jec t suppor ted by The Dan ish Counc i l fo r Independen t Research | Techno logy and Produc t ion Sc iences , M in is t ry o f Sc ience , Innova t ion and H igher Educa t ion

•  To make robo ts soc ia l l y in te l l i gen t and capab le o f es tab l i sh ing durab le re la t ionsh ip w i th the i r users

•  Mul t i -moda l : speech , v i s ion , fac ia l express ion e tc .

DURABLE INTERACTION WITH SOCIALLY INTELLIGENT ROBOTS


•  Fi rs t genera t ion

HARDWARE


05/08/15

5

•  Fi rs t genera t ion

HARDWARE


•  Second genera t ion

HARDWARE

5 A U G U S T 2 0 1 5 A A L B O R G U N I V E R S I T Y 1 0

05/08/15

6

•  Second genera t ion •  What changed?

•  New body mate r ia l and shape •  New ears •  iPad ( inpu t and ou tpu t ) •  New robo t base (P ioneer P3-DX)

HARDWARE


•  Sys tem OS: UBUNTU

•  Robot OS: ROS •  A grea t f ramework fo r each modu le / func t ion to commun ica te

•  Wide ly used and h igh-qua l i t y so f tware ava i lab le

•  Open-source

•  Suppor t Py thon o r C

SOFTWARE


05/08/15

7

SOFTWARE


•  The Day o f Research 2014

DEMOS


05/08/15

8

•  ”S ikker 7 ” in N ibe

DEMOS


•  The Cu l tu re N igh t 2014

DEMOS


05/08/15

9

•  The peop le ’s meet ing 2015

DEMOS


Research •  User mode l ing •  Rein fo rcement fus ion

Co l labora t ion /App l i ca t ion : •  Futu re Nurs ing Home

Poten t ia l app l i ca t ion :

•  Play ing / lea rn ing w i th ch i ld ren

FUTURE WORK


05/08/15

10

1 . I n t r o d u c t i o n t o R o b o t O p e r a t i n g S y s t e m ( R O S ) 2 . I n t r o d u c t i o n t o i S o c i o B o t a n d N A O r o b o t , a n d d e m o s 3 . S o c i a l R o b o t s a n d A p p l i c a t i o n s 4 . M a c h i n e L e a r n i n g a n d P a t t e r n R e c o g n i t i o n 5 . S p e e c h P r o c e s s i n g I : A c q u i s i t i o n o f S p e e c h , F e a t u r e E x t r a c t i o n a n d S p e a k e r L o c a l i z a t i o n 6 . S p e e c h P r o c e s s i n g I I : S p e a k e r I d e n t i f i c a t i o n a n d S p e e c h R e c o g n i t i o n 7 . I m a g e P r o c e s s i n g I : I m a g e A c q u i s i t i o n , P r e - p r o c e s s i n g a n d F e a t u r e E x t r a c t i o n 8 . I m a g e P r o c e s s i n g I I : F a c e D e t e c t i o n a n d F a c e R e c o g n i t i o n 9 . U s e r M o d e l l i n g 1 0 . M u l t i m o d a l H u m a n - R o b o t I n t e r a c t i o n

COURSE OUTLINE
