1

(Jeremy)Behavioral Informatics and Interaction Computation Lab (BIIC)

:

()

(AIRC)2016.11.30

2

PhD: Dec. 2012 @ USCAdvisor: Prof. Shrikanth Narayanan (Shri)

Thesis TitleBehavioral Signal Processing: Computational Approachesfor Modeling and Quantifying Interaction Dynamics inDyadic Human Interactions

2013 2014Identity Protection

Credit Card ProtectionFraud Prevention

2014 BIIC@EE Dept., National Tsing Hua University

3

Seek a window into human mind and traits

through engineering approach

S. Narayanan and P. G. Georgiou, Behavioral signal processing: Deriving human behavioral informatics from speech andlanguage," Proceedings of the IEEE, vol. 101, no. 5, pp. 12031233, 2013.

4

Enabling Technologies

Domain Experts Knowledge

Low level descriptors

Acoustic features

Motion features

Text features

Image features

Speech recognition

Face recognition

Action recognition

Dialog act tagging

Keyword spotting

Text processing

Sentiment Analysis

Affect recognition

Speaker states and

traits

Visual-speech

processing

Interaction modeling

Subjectiveassessment

Internal state & construct

Neuro-developmental disorder

Evidence-based

observational coding

Intervention efficacy

Coder variability

control

Development of coding manual

Self report measure validity

Coding mechanism

Social behavior

Affective behavior

Communicative

behavior

Dyadic behavior

5

Affective Computing

Social Signal Processing

Paralinguistic Recognition

Physiological/Pathological Disorder Recognition/Prediction

6

Behavior signal processing

QUANTITATIVE ()QUANTITATIVE EVIDENCE DIRECTLY FROM MEASURABLE SIGNALS

EFFICIENCY () :HELP DO THINGS THAT EXPERTS KNOW TO DO WELL MORE EFFICIENTLY, CONSISTENTLY & AT SCALE

SUPPLMENTARY ():

COMPLEMENT WITH GOLD STANDARD METHOD WHEN APPROPRIATE

POSSIBILITY ():

TOOLS FOR NOVEL ACTIONABLE INSIGHT DISCOVERY

7

COMPUTING BEHAVIORAL TRAITS & STATES FOR DECISION MAKING & ACTION

...

8

. . .

9

:

10

(20133 13 )

(20135 29 )

11

12

200/

:

?

13

62.5 ():" "

89 ():

14

15

-frame Dense Points Tracking

TRAJ

MBHxy

Each = A Unit-level (66ms) -length Derived Video features

(66) ()

1

2

3

1

2

Acoustic LLDs

Each : = A Unit-level (200ms)-length Dense Acoustic Features

Functionals

1: {1, 1}1

1:1

2:1

:1

1:

(200)

2: {1, 2}

3: {1, 3}

4: {1, 4}

()

16

17

18

Word2Vec

19

Word2Vec

20

...

N-gram K-meansAll Documents

BOWper Document

:

Word2vec

N

21

Support vector regression

Support vector regression

+

Support vector regression

22

1

2

Spearman correlation()

= .

3

23

2

1

2

2

1

10

= .

= .

= .

24

()

25

?

26

()task

Task 1 - feature

Task 2 - feature

Task 8 - feature

.

.

.

Kernel

Multi-task learning

27

?

?

28

?

29

- . . .

30

31

0

2000000

4000000

6000000

8000000

2010 2011 2012 2 0 1 3 2014 2015

2010~2015 THE NUMBER OF EMERGENCY PATIENTS

7,200,000

32

33

(Taiwan Triage and Acuity Scale, TTAS)

34

(=)

(~200)

(=)

(=)

(=)

Raw audio-videorecording

S1

S2

Sk

. . . MFCCPitch

Intensity

1 : [1,1]

2 : [1, 2]

: [1,]

35

:

S1

36

:

Support vector classification

Support vector classification

37

81%

38

(: 0-3, : 4-6, : 7-10)

: :

: :

: :

39

40

. . .

41

social-communicative neurodevelopmental disorder

Prevalence: 1 in 68 children (1 in 42 males) diagnosed [CDC2014]

ASD: Spectrum disorder due to the extreme heterogeneity

BSP in Autism ?

What is Autism?

42

ROLE OF BSP?

ADOS social interactive

?

Analysis at scale

Quantitative evidence from signals

43

Autism Diagnostic Observation Schedule

[Lord 2001]

Subject interacts with a clinician for ~30-45 minutes

Used to help psychologists diagnose autism (current gold standard)

Psychologists are trained using stringent training protocol

28 codes to rate

44

(

()

45

ADOS

46

Can we?

?

47

ADOSEmotion Part

Multimodal Turn-taking Behavior

Coordination Time Series

48

0.81

49

BSP

50

51

Continuing

We can now start imagining the application of this analytics:

(1) Early detection at home (Parent modeling?)(2) Clinician training progress

More?

52

&

+1

53

BiiC: BSP

fMRI

54

In-car

BiiC,

In-home

In-classroom on and on

55

MEASURING & QUANTIFYING HUMAN BEHAVIOR: A CHALLENGING ENGINEERING PROBLEM

Data, Algorithms, Interpretable Behavior Analytics, Actionable Insights

CHALLENGING & PUSHGING the BOUNDARY of STATUS QUO across NUMEROUS FIELDS

Data, Algorithms, Interpretable Behavior Analytics, Actionable Insights

56

57

()

()

Pattern ()

Contextualize

58

59

:

application domain

60

Challenging the status quo/ Pushing scientific boundaryMaking a positive impact

61

BiiC lab @ NTHU EEhttp://biic.ee.nthu.edu.tw

THANK YOU . . .

many COLLABORATORS + the entire BIIC lab