Martin J. Pickering, Simon Garrod, Behavioral and Brain Sciences, 2012.

An Integrated Theory of Language Production and Comprehension

Computer Science & Engineering2012-20835

Sang-Woo Lee

Background - Aphasia

3

Expressive aphasia Agrammatic aphasia Understand what other people say, but cannot speak the sen-

tence well. Caused by damage to, or developmental issues in the anterior

regions of the brain Including (but not limited to) the Broca’s area

Broca’s Aphasia

4

Also known as Receptive Aphasia Fluent apahsia, or sensory aphasia Speak the sentence fluently, but not well-organized sense in

their speech Traditionally associated with neurological damage to Wernicke’s

area in the brain (Actually it is not just simply associated to Wernicke’s area in current exper-

imental result, but anyway…)

Wernicke’s Aphasia

5

There are modules which specify some function

The “classical Lichtheim-Broca-Wernicke” Model

perceptionaction

6

Background - Aphasia Traditional independence of production and comprehension Interweaving in action and action perception

Perception process in action Predict next action of other Joint Action

Interweaving in Production and Comprehension Comprehension process in Production Predict next speech of other Interactive Language

Professor’s Question

Contents

Traditional independence of

production and comprehension

8

Traditional model of communication Discrete stages

A produces, B comprehends B produces, A comprehends

9

Assumes “horizontal split” between pro-duction and comprehension Arrows-within-arrows indicate feedback (in in-

teractive accounts) But this feedback is internal to production or

comprehension It may involve “general knowledge” But production does not involve comprehension pro-

cesses And comprehension does not involve production pro-

cesses

Horizontal Split

10

Interlocutors are not static, as the traditional model as-sumes, but are “moving targets” performing a joint activitiy (Garrod & Pickering, 2009)

Example of predict in Behavioral Instance

11

Example of predict in Neuroscience

Big(neuter)

Painting (neuter)

Big (common)Bookcase (com-

mon)

(Pickering & Garrod, 2007)

große Gemälde großen Bücherschrank

12

Also, many experiments demonstrate ef -fects of one on the other Picture-word interference (Schriefers et al.,

1990) Word identification affected by externally con-

trolled cheek movement (Ito et al., 2009) And strongly overlapping neural circuits

for production and comprehension (e.g., Pulvermüller & Fadiga, 2010; Scott et al., 2009)

Other Counterexample

13

Prediction process could be naturally understood with pro-duction module

When they comprehend the utterance, they also use pro-duction model internally.

Result

- There is forward modelTo predict perception caused by their own ut-terance- Fast alert when you say something wrong

Interweaving in action and action perception

Perception process in action Predict next action of other Joint Action

Interweaving in action and action perception

• Close links between action and action percep-tion, e.g.– participants’ arm movements affected by observing

another person’s arm movements (Kilner et al., 2003)– And making hand movements can facilitate concur-

rent visual discrimination of deviant hand postures (Miall et al., 2006)

• Such links could have various purposes – Supporting overt imitation– facilitating memory or understanding (“postdictively”)

• But authors propose that they aid prediction of own and others’ actions, by use of a forward model– Based on computational neuroscience (Wolpert, 1997;

see Grush, 2004)

Forward modelling in action In our terms, the action command causes the action im-

plementer to move the hand and the perceptual imple-menter to construct the percept

And the efference copy causes the forward action model to generate the predicted hand movement and the forward perceptual model to construct the predicted percept

Forward Modeling in Action- Just act- Feel involved per-

cepts of your own acte.g. Own coordination infoFeeling of wind blow-ing to your armGravity info …

18

Efference Copy

Forward Modeling in Action- Predict perception caused by their own action

e.g. Own coordination infoFeeling of wind blowing to your armGravity info …

Prediction-by-Simulation

- Predict perception of other’s next actionby seeing other’s current action

Joint Action• People are highly adept at joint activities

(Sebanz et al., 2006). – ballroom dancing, playing a duet, carrying a

large object together– Precise timing is crucial

• To succeed, A predicts B’s action and B predicts B’s action

22

Joint Action

Interweaving in Production and Comprehension

Comprehension process in Production Predict next speech of other Interactive Language

Forward modeling in language production

• Action implementer production imple-menter

• Perceptual implementer comprehension implementer

• Action command production command– Drives the production implementer– Efference copy drives the forward models

• Comparator monitor – compares the utterance percept and the pre-

dicted utterance percept

Unifying production and comprehension

• Production and comprehension are inter-woven– Tight coupling in dialogue (Clark, 1996; Picker-

ing & Garrod, 2004)– Behavioural experiments show effects of com-

prehension processes on production and vice versa (e.g., Schriefers et al., 1990)

– Overlap of brain circuits for production and comprehension (e.g., Pulvermuller & Fadiga, 2010)

• Such interweaving facilitates prediction of self and other’s utterances

26

Classical modeling in language production

production command ti

Production implementer

Efference copy

Utterance

tphonsynsemp ,,

Forward production model

Comprehension implementer

Utterance percept tphonsynsemc ,,

Forward comprehension model

monitor

Predicted utterance

tphonsynsemp ,,ˆ

Predicted utterance percept tphonsynsemc ,,ˆ

- Just say utterance,- Listen what you say.

27

Forward modeling in language production

production command ti

Production implementer

Efference copy

Utterance

tphonsynsemp ,,

Forward production model

Comprehension implementer

Utterance percept tphonsynsemc ,,

Forward comprehension model

monitor

Predicted utterance

tphonsynsemp ,,ˆ

Predicted utterance percept tphonsynsemc ,,ˆ

- Predict perception caused by their own ut-terance- Fast alert when you say something wrong

Self-monitoring

Speaker wishes to say kite In the past, she has always constructed

the kite-concept and then uttered /k/ She therefore constructs forward model

p^[phon](t) = /k/ If she then incorrectly constructs p[phon]

= /g/, the monitor notices the mismatch If she believes the forward model, she will

detect an error (and perhaps reformulate) Otherwise, she will alter her forward model

29

Prediction-by-simulation

Efference copy

Forward production model

Comprehension implementer

Inverse model + context

Comprehension implementer

Forward comprehension model

monitor

Covert imitation

Person B

Person A

B’s utterance 1,, tphonsynsemp B

Derived production command

tiB

Predicted utterance

1,,ˆ tphonsynsemp B

Predicted utterance percept 1,.ˆ tphonsynsemc B

Utterance percept tphonsynsemc B,,

Utterance percept 1,, tphonsynsemc B

Overt Responses

Derived production command

1tiB

B’s utterance tphonsynsemp B,,

Derived intentional act of communication iA(t+1)

- Predict perception of other’s next utteranceby listening other’s cur-rent utterance

Big(neuter)

Painting (neuter)

30

Interactive Language

Joint action involves combining accounts of action and action perception

Similarly, interactive language involves combining ac-counts of production and comprehension Facilitates coordination (e.g., short intervals between speak-

ers; Wilson & Wilson, 2005) Facilitates alignment (developing same representations; Pick-

ering & Garrod, 2004) Alignment in turn facilitates comprehension (better prediction

of others)

31

Interactive Language

Conclusion We propose that language production and com-

prehension are interwoven It assumes a central role to prediction in produc-

tion, comprehension, and dialogue Speakers construct forward models to predict as-

pects of their upcoming utterances Listeners covertly imitate speakers and use for-

ward models to predict the speakers Our account helps explain the efficiency of pro-

duction and comprehension and the remarkable fluidity of dialogue

Thank you

Professor’s Question

Comprehension process in Production Predict next speech of other Interactive Language

35

Q1: Give the evidence for how language production and comprehension are tightly interwoven. How does this relate to the perception-action cycle theory of cognitive systems?

Question 1

36

Q2: Explain and give the evidence for how action, action perception, and joint action are interwoven. Explain how the authors use this to develop accounts of production, com-prehension, and interactive language.

Question 2

Action implementer pro-duction implementer

Perceptual implementer comprehension implementer

Action command produc-tion command

Comparator monitor

participants’ arm move-ments affected by ob-serving another person’s arm movements (Kilner et al., 2003)

And making hand move-ments can facilitate con-current visual discrimina-tion of deviant hand pos-tures (Miall et al., 2006)

37

Q3: Give examples of what behavioral and neuroscientific data on language processing can be explained by the integrated theory of language production and comprehension explains, while modular theory does not.

Question 3

- Behavioral data

- Neuroscientific data (Pick-ering & Garrod, 2007)