WP4 Language Emergence - University of Plymouth · 2011. 7. 13. · Katrin Lohan, Lars Schillingmann, Sascha Griffiths, Gerhard Sagerer, BIEL. ... Kerstin Dautenhahn, Yo Sato, Joe

WP4 Language Emergence

Britta Wrede (BIEL)Katharina Rohlfing, Karola Pitsch,

Katrin Lohan, Lars Schillingmann, Sascha Griffiths, Gerhard Sagerer, BIEL

Jun Tani, RIKENStefano Nolfi, CNR

Angelo Cangelosi, Martin Peniak PLYMChrystopher Nehaniv, Kerstin Dautenhahn, Yo Sato, Joe Saunders,

Frank Förster, Caroline Lyon, UHKerstin Fischer, Arne Zeschel, USD

Overview

Task 4.1• Generalization as a basis for emergence

of symbolic systems (start: M7)Task 4.2

• Acoustic Packaging and the learning of words (start: M13)

Task 4.3• From single word lexicons to

compositional languages (start: M13)Task 4.4

• Constructional grounding and primary scenes (start: M19)

Task 4.5• Evolutionary origins of action and

language compositionality (start: M31)

ITALK Year 3 Review Genoa, 21. June 2011

Progress in WP4 in Y3• 4.1 Language learning with MTRNN:

– Generalisation wrt noise, syntactic category, sentence complexity

• 4.2 Acoustic Packaging / Synchrony– Synchrony: from on- and offsets towards more fine-grained

synchrony– Results on sync between syntax and motion

• 4.3 Word order facilitates– Learning syntactic categories and related actions– And to generalise

• 4.5 Origins of compositionality– Action compositionality to learn language compositionality in

interaction

• 4.4 Construction learning– Grammar induction– Higher level grounding in infants: Semantic salience (internal

reasoning) and input frequency (tutor input)

ITALK Year 3 Review Genoa, 21 June 2011

ActionActionHierarchy 4.1

AcousticPackages 4.2

Objectives & Goals

Speech

GrammaticalConstructions

4.3

4.4

Lexicon Construction


Evolutionary Originsof Compositionality 4.5



Objectives & Goals

Speech


4.3

4.4



4.1 Generalization as a basis forthe emergence symbolic system

RIKEN


Objective

• Learning of complex sentences by MTRNN

• Analysis on effects of noise in generation and learning.

• Analysis on internal trajectories of representing sentences

W. Hinoshita, H. Arie, J. Tani, H.G. Okuno, T. Ogata: "Emergence of hierarchical structure mirroring linguistic composition in a recurrent neural network", Neural Networks, Vol.24, pp.311-320, 2011


MTRNN model

IO vector ( 30 dimensions) 6 dimensional parametric vector P( = Cs Initial states)

IO : 30τ = 2

□a b z ,. ?c ・・

・

・

Cf : 40τ = 5

Cs: 11τ = 70

( = white space )

Example for training

• Verb

• Noun : ball, box• Article : a, the• Adverb : slowly, quickly• Adjective

intransitive : run, walk, jumptransitive : touch, kick, punch

size : big, smallcolor : yellow, blue, red

7 categories, 17 words

Experimental procedure1. 100 sentences are generated from the CFG．2. 80 out of 100 are trained for MTRNN.3. Evaluate MTRNN recognition capability by utilizing

the 100 sentences.

a. Recog：Sentence Cs Init Vectorb. Gene：Cs Init Vector sentencec. Compare：Target sentence in (a) and the generated

one in (b)

Eval

If correctly generated, adequate structure is self-organized.

-60 -50 -40 -30 -20 -10 0 10-15

-10

-5

0

5

10

15

Cs: Initial State Analysis

PCA1

PCA

2

Simplicity

Simplicity in objective phrase

No adjectiveEx: Punch a box.

Two adjectivesEx: Kick a big red ball. One adjective

Ex: Touch a red ball.

adverb

Yes

NoIntransitiveEx: Walk slowly.

Summary

• Functional hierarchy: Alphabet, word, sentence

• Generalization capability: – Recognize unlearned sentences

• Auto-correction of incorrect sentences– Positive effects by learning through noisy inputs

• Sentence structures appear in init state space of slow context units.– Complexity of objective phrases (w/wo adjectives)– w/wo adverbs– Intransitive versus transitive sentences


4.2 Acoustic Packaging

Bielefeld University


Acoustic Package

Acoustic Package

t

t

Speech

Motion

A Computational Model ofAcoustic Packaging

Goals

• Acoustic packaging makes use of interaction between modalities at an early processing level

• Evaluation showed AP is able to reflect differences between adult-adult and adult-child interaction

• Goals Y3 + Y4– Using Acoustic Packaging for word learning– Providing learning units for further processes– Generating feedback events


Cues Focusing on Details in the Interaction

Cues Especially Relevant for Temporal Action Segmentation

Additional Cues and their Role in Acoustic Packaging

Acoustic Packaging

Speech Activity

Motion Segmentation

Finding emphasized syllables:Acoustic Prominence

Trajectory extraction:Motion/Color Saliency

Detecting Moving Colored Objects

•Detecting changing regions

•Clustering in YUV color space

•Ranking according to color distance (U,V) to centroid of all clusters

•Heuristical filtering•Trajectory accumulation


Acoustic Prominence

• Goal: Relative ranking of syllables emphasis within an utterance

• Syllable Segmentation– Mermelstein algorithm

• Features for Emphasis Rating[Tamburini, Wagner 2007]

– Nucleus duration– Spectral emphasis– Pitch movements– Overall intensity

• Currently used: Spectral emphasis• Example with 3 syllables context

Und zum Schluss packen wir noch den roten Becher in den gelben Becher

Finally we put the red cup in the yellow cup

Prominence

roten

Feedback based on Acoustic Packages

• iCub replays prominent syllables

• iCub replays trajectories• Acoustic packages simplify

access to corresponding multimodal events at a time

• Example 1 (implemented)– iCub replays syllables

associated with a specific cup color triggered by showing the cup to iCub

• Example 2 (in progress)– iCub replays trajectories

associated with a specific (prominent) syllable triggered by speech

Acoustic Package

Speech Interval

Motion Segments

Trajectories

Syllable Segmentation

Prominence Ranking


Prominent Syllable –Trajectory Color Association

(en)


iCub‘s Perspective

Synchrony betweenverbal utterances and action

• Semantic: stressed color label (“red”, “yellow”, “green”) and color of detected trajectory

• Temporal: distance of prominent syllable to nearest max trajectory change


Synchrony betweenverbal utterances and action (USD, BIEL)

• Results: Sig. Correlations between– Syntactic category and prominence level– Syntactic category and movement velocity– Pause length and movement velocity

• Further Questions– Synchrony inside vs outside AP– Syntactic structure inside vs outside AP


4.3 From single word lexicons to compositional languages

University of Plymouth


Word Order

• As a structural cue for– category information, e.g. the N, look at the N– semantic roles, e.g. John kisses Mary

• children are sensitive to such cues (e.g.Gomez 2007)

• children use this information for learning (e.g. StClair et al. 2010)


Word Order Learning

Adjective – Noun Constructionstouch green ball touch green cubetouch red balltouch red cube

Hypothesis: Word order providesinformation about

grammatical category (adjective - noun)semantic category (colour - shape)


Recurrent Neural Network

Arm joint / Touch / Shape / Colour

5 Inputs -language

Arm joint / Touch / Shape / Colour

5 Inputs -language

10 Hidden units

5 Inputs – rec. Lang.


• The robot is trained with only objects and colours in isolation

Experiment I: Synonyms

By analysing internal representations no distinction between name and colours are observed

t1: “TOUCH” input t3: Action

TOUCH BALLTOUCH GREENTOUCH CUBETOUCH RED

Input for Word Order Learning

• Words are input to the neural network in sequence

“TOUCH BALL” (t1) TOUCH(t2) BALL

“TOUCH GREEN” (t1) TOUCH(t2) GREEN

“TOUCH RED CUBE” (t1) TOUCH(t2) GREEN(t3) CUBE


Training

• Neural Network trained with BPTT algorithm

• Training is made of sequences of words followed by the action of the robot (touch or not touch the object)

• Robot experiences both “correct” and “uncorrect” sentences:

Environment Language input Action

Red ball “TOUCH RED BALL”

Touch the ball

Green cube “TOUCH RED CUBE”

Not touch the ball


EXPERIMENT IIWord order

Environment Language input Expected action

Red cube “TOUCH GREEN” Not touch the cube

Red cube “TOUCH GREEN CUBE”

Not touch the cube

Green cube “TOUCH GREEN” Touch the cube

Green ball “TOUCH GREEN BALL” Touch the ball

Example of training sentences providing word order info:

After the training the robot was tested with additional utterances not originally included in the training set.

RESULTS:The robot is able to perform the related action and

correctly generalises to novel sentences


• Internal representations change their structureaccording to the word sequence experienced in input

Prediction: t2: {COLOUR} input Prediction: t3: {OBJECT} input

Ball

Cylinder

Cube


EXPERIMENT IIWord order

EXPERIMENT IIINew colour

• Generalisation to a new linguistic category was tested

• A new colour adjectives(blue) was included in the vocabulary

• The robot was tested withoutadditional training onlanguage (blue colour wasexperienced by the robot)

RESULTS: successful generalisation to new colour adjective


• the new colour adjective was successfullyclassified as other colour termsthe unseen input was correctly classified only on the basis of the word

orderthe distributional cue from the word order was thus successfully

interpreted as a clue to the semantic category (colour)

Self-organised Internal representation


EXPERIMENT IIINew colour

4.5 Origins of compositionality

University of Plymouth


Hypothesis

Exposing the robots to a task and an environment that present compositional features, such as

Move A Move BGreen ball Red ball Red cube

could lead the evolutionary process to organise the robot’s knowledge in a compositional structure.

This structure, in turn, could guide the communication system toward the evolution of compositional features.


Precondition

• Communication is grounded in agents’ sensorimotor coordination.

• Interaction between two robots• Collective task


Methodology

Evolutionary robotics

Neural network as control system Genetic algorithm to

evolve synaptic weights

Fittest reproduce


Experimental scenario

3 objects 3 pre-evolved primitive behaviours (touch, grasp, lift)

Communication channels allow robots to exchange signals

Robots are evolved to perform the same action on the same object

A

B


Task 4.4 Constructional grounding

University of Hertfordshire


• Hypotheses: • Children first learn object words (e.g. nouns) and then predicate words

(e.g. verbs) • Object bias: the first phonological words learnt assumed to be of ‘entity’

type• we assume prosodically salient words get associated with objects

• Approach: Semantic bootstrapping idea:• Classify lexicon into predicates and entities, and assign syntactic

constraints• Iterate hypothesis and revise lexicon• Models two-word stage (18-24 months)

o Results: Simulation o Using corpus from Kaspar studyo Result: approx. 65% of the words correctly classified

o Ongoing: interactive learning in the iCub studies with robot producing two word utterances

Grammar induction with simple semantic types


Review Meeting 2 St.Albans, UK

Semantics-Based Grammar Induction: Example

Lexicon state (before):

• Now facing an utterance, e.g. “You see that red big star, Kaspar”

↑ ↑ ↑phonologically known also referentially known

• Classify the words into two types: predicates and entitieso The learner already knows [star] belongs to the arg set, so

possible partitions are (predicates/entities):{ you } / { see, star } (wrong); { you, see } / { star } (wrong); { see } / { you, star } (correct)

• Register the results in lexicon: suppose the second (wrong) hypothesis is taken

Lexicon state (after):


Task 4.4 Constructional grounding

University of Southern Denmark


Two types of language grounding

• Experiential grounding

– Linking ‘elementary symbols’ to the world– aka ‘direct grounding’

• Higher-level symbol grounding

– Linking symbols to symbols (non-elementary to elementary)

– aka ‘grounding transfer’ (Pezzulo et al. 2011)


Two dimensions of transfer

• Learning new forms through tutoring

– Tutor (says): ‘An X is a Y that is Z’(Harnad 1990; Cangelosi and Riga 2006)

– Learning new symbols from known symbols:associate known meaning with new form

• Learning new meanings through reanalysis

– Learner (reasons): ‘X2 is a special type of X1’(Johnson 1999)

– Learning new uses of a symbol from known uses of the same symbol: associate known form with new meaning


Constructional grounding theory

• Johnson (1999)

– “a sign that is relatively easy for children to learn (the sourceconstruction) serves as the model for another more difficultsign (the target construction), because it occurs in contextsin which it exemplifies important properties of that sign in a way that is especially accessible to children”(Johnson 1999: 1)


‘Accessibility’

• Two potentially conflicting determinants:

– Quantitative salience: input frequency

Prediction: developmentally basic variants of a construction are acquired first because

they are (more) common in the input

– Qualitative salience: semantic concreteness

Prediction: developmentally basic variants of a construction are acquired first because

they are semantically (more) concrete


Case study: possession

• Prenominal possessives: [NP‘s NP]

– Developmentally basic: one of the earliest relation-ships marked in emerging child multiword speech(Lieven, Salomo & Tomasello 2009)

– Polyfunctional: wide semantic space (many different „meanings“)

– Semantically asymmetric: some meanings directlygrounded, others considerably more abstract

– Potential for reanalysis: Interpretive overlap betweenvariants


Semantic space

• Some relations marked bypossessives...

– Privileged access John’s car– Partonomy John’s hand– Kinship relations John’s brother– Property ascription John’s impatience– Creation John’s poem– Disposal John’s train (= the one he‘s on)– Participant–event John’s arrival– Location England’s cathedrals– Temporal setting–event Last year’s meeting– ...


concrete

abstract

Study overview

• Aims: For each type of possessive relation,

– What is the order of acquisition of relevant variants?– Which operationalisation of ‘accessibility’ is the more

powerful predictor of acquisition order?

• Data: 3 longitudinal CHILDES corpora– Brown corpus; target child: Sarah; age range: 2;3-5;1– Kuczaj corpus; target child: Abe; age range:2;4-5;0– Sachs corpus; target child: Naomi; age range: 1;2-5;1

• Method: Rank order correlations (Kendall)– Acquisition order vs. concreteness/frequency ranks


Data

• Overview:

– Automatic extraction (all N+N sequences in entire corpus)

– Plus 10 lines anterior and 5 lines posterior context

– Full manual coding in context

– Coded for: type & subtype of relation (concreteness rank)

Child N+N utterances coded Hits (caretaker) Hits (child) Total hitsSarah (Brown) 3675 399 101 500Abe (Kuczaj) 4509 90 175 265

Naomi (Sachs) 1674 180 141 321


Sample results: Abe

• Inherent possessives: age vs. concreteness

Kendall’s τ =.40, p=.53

Category Example Concreteness AcquiredPERSON – BODY PART Dad’s nose 1 3ANIMATE – BODY PART the skunk’s tail 2 2

PERSON – ATTRIBUTE/EXPERIENCE Mummy's name 3 1PERSON – EVENT my uncle’s funeral 4 4

SETTING/MEASURE – EVENT last year’s State Fair 5 5


Results

• For 2 of the 3 learners (Abe & Naomi),

– Sig. Cor: Input frequency with order of acquisition: the most common uses of the construction were learnedfirst

– Sig Cor: Input and output frequency

• For the third learner (Sarah),– Neither input frequency nor concreteness correlated with

order of acquisition

⇒Concreteness alone failed to yield a significant effect⇒Neither factor could fully explain the observed

orders alone


Conclusion

• ‘Quantitative salience’ outperforms ‘qualitative salience’ as a predictor of ease of acquisition (for our data)

multi-factorial model necessary to explain order of acquisition

Outlook• Corroboration by results from denser corpora/other

constructions desirable

• Computational implementation (i.e. grounded robotic learning of prenominal possessive cxns) currently underway (Zeschel& Tuci to appear)


WP 4 Summary Y3• 4.1 Language learning with MTRNN:

– Generalisation wrt noise, syntactic category, sentence complexity

• 4.2 Acoustic Packaging / Synchrony– Synchrony: from on- and offsets towards more fine-grained

synchrony– Tentative results on sync between syntax and motion


• 4.5 Origins of compositionality– Action compositionality to learn language compositionality in

interaction

• 4.4 Construction learning– Grammar induction– Higher level grounding in infants: Semantic salience (internal

reasoning) and input frequency (tutor input)


Evolutionary OriginsOf Compositionality 4.5



Summary

Speech


4.3

4.4



ITALK Year 1 Review Düsseldorf, 1 July 2009

：Recognition of incorrect sentences

Evaluate auto-correction capability

Walk slowly.

TargetWals slowly.

incorrectnoise

比較 Init Vector

…

…

…

…

…

…

Error backpropagation

Walk slowly.

Generated…

…

…

…

…

…

Forward Calculation

Noise addition probability and Success Rate

50.0

55.0

60.0

65.0

70.0

75.0

80.0

85.0

90.0

95.0

100.0

0 5 10 20 30 40

CLEAN

NOISE

(%)

Acc

urac

y R

ate

(%)

ρ (Noise addition

MTRNN Activation pattern (example)

Cs

Cf

IO

Step

p u n c h □ t h e □ s m a l l □ y e l l o w □ b o x □ s l o w l y .

-2-1.5-1-0.5-0.5

0

0.5

1

PCA2PCA1

PCA3

punch the

yellow

slowlybo

x

-1-0.500.5

Punch the yellow box slowly.

-2-1.5-1-0.5-0.5

0

0.5

1

PCA2PCA1

Kick a small yellow ball.

PCA3

kick a

small

yellow

ball

-1-0.500.5

Cs: Transition of Slow Context Activation

Semantics-Based Grammar Induction: Results so far

• Crucial question is whether type hypotheses convergeo Simulation with the past experimental data (Year 1) suggest they doo Type mapping results: object bias improves learning with t-test significance

at p=.0035

• We are now testing it in an HRI experiment, with iCub actually responding to the partitipants with two-word utterances

F-score (P: precision, R: recall)

Baseline (no bias)Semantic bootstrapping with object bias

.4675 (P:.4502, R:.4863)

.6417 (P:.6190, R:.6663)


Overview• 4.1 Language learning with MTRNN:

– Generalisation wrt noise, syntactic category, sentencecomplexity

• 4.2 Acoustic Packaging / Synchrony– Synchrony: from on- and offsets towards more fine-

grained synchrony– Tentative results on sync between syntax and motion


• 4.4 Grammar Induction– Incremental lexicon development

• 4.5 Origins of compositionality– Action compositionality to learn language

compositionality in interaction

InternalReasoning

Input & InternalReasoning

InternalReasoning

Input &InternalReasoning


Motivation –Action Segmentation in Infants

• Children need to discover meaningful action units– Language helps to divide a

sequence of events into units– Prerequisite: synchrony between

language and events

• Described as acoustic packaging (AP) [Hirsh-Pasek and Golinkoff 1996]

• AP can provide a bottom-up action segmentation


Acoustic Package

Acoustic Package

t

t

Speech

Motion

A Computational Model ofAcoustic Packaging

• Segmentation of input cues– Acoustic temporal segmentation– Visual temporal segmentation

• Cue fusion– Temporal association of multi-

modal input streams– Results of the association process

are Acoustic Packages

/a:/ /p/ /h/ /2:/[noise1] /d/ /a/ /n/

Collection: Syllables From Multiple Runs


4.2 Summary

Summary

• Acoustic packaging provides a bottom-up action segmentation• Acoustic packaging makes use of interaction between modalities at an

early level• Acoustic packages simplify access to corresponding multimodal events at

a time• Synchrony between action and syntactic structure

Long term goals• Understanding actions• Language learning

Next Targets

• Consolidation of acoustic packages e.g. clustering, selection of relevant packages

• More complex Feedback


Slide Number 1OverviewProgress in WP4 in Y3Objectives & GoalsObjectives & Goals4.1 Generalization as a basis for the emergence symbolic systemObjectiveMTRNN modelExample for training Experimental procedureCs: Initial State AnalysisSummary4.2 Acoustic PackagingA Computational Model of�Acoustic PackagingGoalsAdditional Cues and their Role in Acoustic PackagingDetecting Moving Colored ObjectsAcoustic ProminenceFeedback based on �Acoustic PackagesProminent Syllable – Trajectory Color Association (en)iCub‘s Perspective�Synchrony between �verbal utterances and action Synchrony between �verbal utterances and action (USD, BIEL) 4.3 From single word lexicons to compositional languages Word OrderWord Order LearningRecurrent Neural NetworkExperiment I: SynonymsInput for Word Order LearningTrainingEXPERIMENT II� Word orderEXPERIMENT II� Word orderEXPERIMENT III�New colourEXPERIMENT III�New colour4.5 Origins of compositionality HypothesisPreconditionMethodologyExperimental scenarioSlide Number 40Slide Number 41Semantics-Based Grammar Induction: ExampleSlide Number 43Two types of language groundingTwo dimensions of transferConstructional grounding theory‘Accessibility’Case study: possessionSemantic spaceStudy overviewDataSample results: AbeResultsConclusion WP 4 Summary Y3SummarySlide Number 57：Recognition of incorrect sentencesNoise addition probability and Success RateSlide Number 60Cs: Transition of Slow Context Activation Semantics-Based Grammar Induction: Results so farOverviewMotivation – �Action Segmentation in InfantsA Computational Model of�Acoustic PackagingCollection: Syllables From Multiple Runs4.2 Summary

Documents

WP4 Language Emergence - University of Plymouth · 2011. 7. 13. · Katrin Lohan, Lars Schillingmann, Sascha Griffiths, Gerhard Sagerer, BIEL. ... Kerstin Dautenhahn, Yo Sato, Joe