Embed Size (px)
Citation preview
05/08/15
1
AAU SUMMER SCHOOL
PROGRAMMING SOCIAL ROBOTS FOR HUMAN INTERACTION
L E C T U R E 1 0 M U LT I M O D A L H U M A N - R O B O T I N T E R A C T I O N
1 . I n t r o d u c t i o n t o R o b o t O p e r a t i n g S y s t e m ( R O S ) 2 . I n t r o d u c t i o n t o i S o c i o B o t a n d N A O r o b o t , a n d d e m o s 3 . S o c i a l R o b o t s a n d A p p l i c a t i o n s 4 . M a c h i n e L e a r n i n g a n d P a t t e r n R e c o g n i t i o n 5 . S p e e c h P r o c e s s i n g I : A c q u i s i t i o n o f S p e e c h , F e a t u r e E x t r a c t i o n a n d S p e a k e r L o c a l i z a t i o n 6 . S p e e c h P r o c e s s i n g I I : S p e a k e r I d e n t i f i c a t i o n a n d S p e e c h R e c o g n i t i o n 7 . I m a g e P r o c e s s i n g I : I m a g e A c q u i s i t i o n , P r e - p r o c e s s i n g a n d F e a t u r e E x t r a c t i o n 8 . I m a g e P r o c e s s i n g I I : F a c e D e t e c t i o n a n d F a c e R e c o g n i t i o n 9 . U s e r M o d e l l i n g 1 0 . M u l t i m o d a l H u m a n - R o b o t I n t e r a c t i o n
COURSE OUTLINE
5 A U G U S T 2 0 1 5 A A L B O R G U N I V E R S I T Y 2
05/08/15
2
“ IN THE CONTEXT OF HUMAN–COMPUTER INTERACTION, A MODALITY IS THE CLASSIF ICATION OF A S INGLE INDEPENDENT CHANNEL OF SENSORY INPUT/OUTPUT BETWEEN A COMPUTER AND A HUMAN. A SYSTEM IS DESIGNATED UNIMODAL IF IT HAS
ONLY ONE MODALITY IMPLEMENTED, AND MULTIMODAL IF IT HAS MORE THAN ONE. ”
KARRAY, FAKHREDDINE, ET AL. "HUMAN-COMPUTER INTERACTION: OVERVIEW ON STATE OF THE ART." (2008).
MULTIMODAL INTERACTION – WHAT?
5 A U G U S T 2 0 1 5 A A L B O R G U N I V E R S I T Y 3
• Extended func t iona l i t y, e .g . we can speak to the robo t ins tead o f t yp ing
• Human-Human l i ke commun ica t ion
MULTIMODAL INTERACTION – WHY?
5 A U G U S T 2 0 1 5 A A L B O R G U N I V E R S I T Y 4
05/08/15
3
Robus tness aga ins t no ise • Data f rom one moda l i t y m igh t be ve ry no isy, however the res t a re
no t – comb ine the moda l i t i es
• Person Iden t i f i ca t ion : background mus ic i s co r rup t ing recorded speech , however v i s ion i s una l te red .
• Speech Recogn i t ion : background mus ic i s co r rup t ing recorded speech , however v i s ion can be used to recogn ize l i p movements and c lass i f y words
• How to know wh ich moda l i t y to “ t rus t ”?
MULTIMODAL INTERACTION – WHY?
5 A U G U S T 2 0 1 5 A A L B O R G U N I V E R S I T Y 5
• Prov ide new in fo rmat ion , wh ich cou ld no t be p rov ided by ind iv idua l moda l i t i es
• Combina t ion o f sound + fac ia l express ion = emot ion
• Learn ing : • Somet imes on ly one moda l i t y i s ava i lab le , bu t no isy
• Use knowledge f rom one moda l i t y to re - t ra in /adap t mode l i n o ther doma in
• Examples : Person Iden t i f i ca t ion , D i rec t ion o f A t ten t ion
MULTIMODAL INTERACTION – WHY?
5 A U G U S T 2 0 1 5 A A L B O R G U N I V E R S I T Y 6
05/08/15
4
• iSoc ioBot
• Research p ro jec t suppor ted by The Dan ish Counc i l fo r Independen t Research | Techno logy and Produc t ion Sc iences , M in is t ry o f Sc ience , Innova t ion and H igher Educa t ion
• To make robo ts soc ia l l y in te l l i gen t and capab le o f es tab l i sh ing durab le re la t ionsh ip w i th the i r users
• Mul t i -moda l : speech , v i s ion , fac ia l express ion e tc .
DURABLE INTERACTION WITH SOCIALLY INTELLIGENT ROBOTS
5 A U G U S T 2 0 1 5 A A L B O R G U N I V E R S I T Y 7
• Fi rs t genera t ion
HARDWARE
5 A U G U S T 2 0 1 5 A A L B O R G U N I V E R S I T Y 8
05/08/15
5
• Fi rs t genera t ion
HARDWARE
5 A U G U S T 2 0 1 5 A A L B O R G U N I V E R S I T Y 9
• Second genera t ion
HARDWARE
5 A U G U S T 2 0 1 5 A A L B O R G U N I V E R S I T Y 1 0
05/08/15
6
• Second genera t ion • What changed?
• New body mate r ia l and shape • New ears • iPad ( inpu t and ou tpu t ) • New robo t base (P ioneer P3-DX)
HARDWARE
5 A U G U S T 2 0 1 5 A A L B O R G U N I V E R S I T Y 11
• Sys tem OS: UBUNTU
• Robot OS: ROS • A grea t f ramework fo r each modu le / func t ion to commun ica te
• Wide ly used and h igh-qua l i t y so f tware ava i lab le
• Open-source
• Suppor t Py thon o r C
SOFTWARE
5 A U G U S T 2 0 1 5 A A L B O R G U N I V E R S I T Y 1 2
05/08/15
7
SOFTWARE
5 A U G U S T 2 0 1 5 A A L B O R G U N I V E R S I T Y 1 3
• The Day o f Research 2014
DEMOS
5 A U G U S T 2 0 1 5 A A L B O R G U N I V E R S I T Y 1 4
05/08/15
8
• ”S ikker 7 ” in N ibe
DEMOS
5 A U G U S T 2 0 1 5 A A L B O R G U N I V E R S I T Y 1 5
• The Cu l tu re N igh t 2014
DEMOS
5 A U G U S T 2 0 1 5 A A L B O R G U N I V E R S I T Y 1 6
05/08/15
9
• The peop le ’s meet ing 2015
DEMOS
5 A U G U S T 2 0 1 5 A A L B O R G U N I V E R S I T Y 1 7
Research • User mode l ing • Rein fo rcement fus ion
Co l labora t ion /App l i ca t ion : • Futu re Nurs ing Home
Poten t ia l app l i ca t ion :
• Play ing / lea rn ing w i th ch i ld ren
FUTURE WORK
5 A U G U S T 2 0 1 5 A A L B O R G U N I V E R S I T Y 1 8
05/08/15
10
1 . I n t r o d u c t i o n t o R o b o t O p e r a t i n g S y s t e m ( R O S ) 2 . I n t r o d u c t i o n t o i S o c i o B o t a n d N A O r o b o t , a n d d e m o s 3 . S o c i a l R o b o t s a n d A p p l i c a t i o n s 4 . M a c h i n e L e a r n i n g a n d P a t t e r n R e c o g n i t i o n 5 . S p e e c h P r o c e s s i n g I : A c q u i s i t i o n o f S p e e c h , F e a t u r e E x t r a c t i o n a n d S p e a k e r L o c a l i z a t i o n 6 . S p e e c h P r o c e s s i n g I I : S p e a k e r I d e n t i f i c a t i o n a n d S p e e c h R e c o g n i t i o n 7 . I m a g e P r o c e s s i n g I : I m a g e A c q u i s i t i o n , P r e - p r o c e s s i n g a n d F e a t u r e E x t r a c t i o n 8 . I m a g e P r o c e s s i n g I I : F a c e D e t e c t i o n a n d F a c e R e c o g n i t i o n 9 . U s e r M o d e l l i n g 1 0 . M u l t i m o d a l H u m a n - R o b o t I n t e r a c t i o n
COURSE OUTLINE
5 A U G U S T 2 0 1 5 A A L B O R G U N I V E R S I T Y 1 9
