Speech Perception in Infant and Adult Brains Colin Phillips Cognitive Neuroscience of Language Laboratory Department of Linguistics University of Maryland

Speech Perception in Infant and Adult Brains

Colin PhillipsCognitive Neuroscience of Language Laboratory

Department of LinguisticsUniversity of Maryland

Overview of Talks

1. The Unification Problem

2. Building Syntactic Relations

3. Abstraction: Sounds to Symbols

4. Linguistics and Learning

In-situ

600

700

800

900

1000

1100

1 2 3 4 5 6 7 8

Region

Reading Time

DeclC

QP

どの生徒に…

with help from ...University of Maryland

Shani AbadaSachiko Aoshima

Daniel Garcia-PedrosaAna Gouvea

Nina KazaninaMoti LiebermanLeticia PablosDavid PoeppelBeth RabbinSilke Urban

Carol Whitney

University of Delaware

Evniki EdgarBowen HuiBaris KabakTom Pellathy

Dave SchneiderKaia Wong

Alec Marantz, MITElron Yellin, MIT

National Science FoundationJames S. McDonnell Foundation

Human Frontiers Science ProgramJapan Science & Technology Program

Kanazawa Institute of Technology

Sensory Maps

Internal representations of the outside world. Cellular neuroscience has discovered a great deal in this area.

Encoding of Symbols: Abstraction

• But most areas of linguistics (phonology, morphology, syntax, semantics) are concerned with symbolic, abstract representations,

...which do not involve internal representations of dimensions of the outside world.

…hence, the notion of sensory maps does not get us very far into language

Outline

• Categories & Abstraction• Speech Perception in Infancy• Electrophysiology: Mismatch Paradigm• Speech Discrimination in Adult Brains• Speech Categorization in Adult Brains• Conclusion

A Category

Another Category

3

III

Categories for Computation

• Membership in a category like “bird” is a graded property

• Membership in a category like “three” is an all-or-nothing property

• “Three” can be part of a symbolic computation

• “Bird” cannot

Abstraction

• Benefits of abstraction– representational economy

– representational freedom

– allow combinatorial operations

• Costs of abstraction– distant from experience - impedes learning

– distant from experience - impedes recognition

Phonetic vs. Phonological Categories

• Phonetic category membership is graded• Phonological category membership is an

all-or-nothing property: all members are equal

• Phonological categories are the basis of storage of lexical forms

• Phonological categories participate in a wide variety of combinatorial computations

Timing - Voicing

Voice Onset Time (VOT)

60 msec

Perceiving VOT

‘Categorical Perception’

Discrimination

Same/Different

Discrimination

Same/Different0ms 60ms

Discrimination


Same/Different

Discrimination



Discrimination



Same/Different

Discrimination




Discrimination




Why is this pair difficult?

Discrimination





(i) Acoustically similar?

(ii) Same Category?

Discrimination





(i) Acoustically similar?

(ii) Same Category?

A More Systematic Test

Discrimination





0ms

20ms

40ms

20ms

40ms

60ms

Discrimination





0ms

20ms

40ms

20ms

40ms

60ms

D T

D

T T

D

Within-Category Discrimination is Hard

Cross-language Differences

R L


R L

R L

Cross-Language Differences

English vs. Japanese R-L


English vs. Hindi

alveolar [d]

retroflex [D] ?

Outline


Development of Speech Perception

• Unusually well described in past 30 years

• Learning theories exist, and can be tested…

• Jakobson’s suggestion: children add feature contrasts to their phonological inventory during development

Roman Jakobson, 1896-1982Kindersprache, Aphasie und allgemeine Lautgesetze,

1941

Developmental Differentiation

0 months 6 months 12 months 18 months

UniversalPhonetics

Native Lg.Phonetics

Native Lg.Phonology

#1 - Infant Categorical Perception

Eimas, Siqueland, Jusczyk & Vigorito, 1971

Discrimination





0ms

20ms

40ms

20ms

40ms

60ms

D T

D

T T

D


English VOT Perception

To Test 2-month olds

Not so easy!

High Amplitude Sucking

Eimas et al. 1971

General Infant Abilities

• Infants’ show Categorical Perception of speech sounds - at 2 months and earlier

• Discriminate a wide range of speech contrasts (voicing, place, manner, etc.)

• Discriminate Non-Native speech contrastse.g., Japanese babies discriminate r-le.g., Canadian babies discriminate d-D

Universal Listeners

• Infants may be able to discriminate all speech contrasts from the languages of the world!

How can they do this?

• Innate speech-processing capacity?

• General properties of auditory system?

What About Non-Humans?

• Chinchillas show categorical perception of voicing contrasts!

#2 - Becoming a Native Listener

Werker & Tees, 1984

When does Change Occur?

• About 10 months

Janet Werker

U. of British ColumbiaConditioned Headturn Procedure


• Hindi and Salishcontrasts testedon English kids

Janet Werker

U. of British ColumbiaConditioned Headturn Procedure

What has Werker found?

• Is this the beginning of efficient memory representations (phonological categories)?

• Are the infants learning words?

• Or something else?

6-12 Months: What Changes?

Structure Changing

Patricia KuhlU. of Washington

#3 - What, no minimal pairs?

Stager & Werker, 1997

A Learning Theory…

• How do we find out the contrastive phonemes of a language?

• Minimal Pairs

Word Learning

• Stager &Werker 1997

‘bih’ vs. ‘dih’and‘lif’ vs. ‘neem’

Word learning results

• Exp 2 vs 4

Why Yearlings Fail on Minimal Pairs

• They fail specifically when the task requires word-learning

• They do know the sounds

• But they fail to use the detail needed for minimal pairs to store words in memory

• !!??

One-Year Olds Again

• One-year olds know the surface sound patterns of the language

• One-year olds do not yet know which sounds are used contrastively in the language…

• …and which sounds simply reflect allophonic variation

• One-year olds need to learn contrasts

Maybe not so bad after all...

• Children learn the feature contrasts of their language

• Children may learn gradually, adding features over the course of development

• Phonetic knowledge does not entailphonological knowledge

Roman Jakobson, 1896-1982

Summary of Development

• Surface10 months

• Memory18 months

MemoryMemory

SurfaceSurface

PhoneticPhoneticAuditoryAuditory ArticulatoryArticulatory

LexicalLexical

Innate

Constructed

Outline


KIT-Maryland MEG System

(160 SQUID detectors)

Brain Magnetic Fields (MEG)

Brain Magnetic Fields (MEG)

SQUID detectors measure brainmagnetic fields around 100 billiontimes weaker than earth’s steadymagnetic field.

160 SQUIDwhole-headarray

pickup coil & SQUIDassembly

It’s safe…

V

skull

CSF

tissue

MEG

EEGB

- noninvasive measurement- direct measurement.

scalprecordingsurface

currentflow

orientationof magnetic field

Origin of the signal

How small is the signal?10-4

10-5

10-6

10-7

10-8

10-9

10-10

10-11

10-12

10-13

10-14

10-15

Earth field

Urban noise

Contamination at lung

Heart QRS

MuscleFetal heart

Spontaneous signal (-wave)

Signal from retina

Intrinsic noise of SQUID

Inte

nsity

of

mag

netic

si

gnal

(T)

Evoked signal

Biomagnetism

EYE (retina) Steady activity Evoked activity

LUNGS Magnetic contaminants

LIVER Iron stores

FETUS Cardiogram

LIMBS Steady ionic current

BRAIN (neurons) Spontaneous activity Evoked by sensory stimulation

SPINAL COLUMN (neurons) Evoked by sensory stimulation

HEART Cardiogram (muscle) Timing signals (His Purkinje system)

GI TRACK Stimulus response Magnetic contaminations

MUSCLE Under tension

Electroencephalography (EEG/ERP)

Event-Related Potentials (ERPs)

s1 s2 s3

John is laughing.

Mismatch ResponseX X X X X Y X X X X Y X X X X X X Y X X X Y X X X...

Mismatch ResponseX X X X X Y X X X X Y X X X X X X Y X X X Y X X X...

Mismatch Response

Latency: 150-250 msec.Localization: Supratemporal auditory cortexMany-to-one ratio between standards and deviants

X X X X X Y X X X X Y X X X X X X Y X X X Y X X X...

Localization of Mismatch Response

(Phillips, Pellathy, Marantz et al., 2000)

Basic MMN elicitation

©Risto Näätänen

Basic MMN elicitation

MMN P300 Näätänen et al. 1978

MMN Amplitude Variation

Sams et al. 1985

Tiitinen et al. 1994

How does MMNlatency, amplitudevary with frequencydifference?

1000Hz tone std.

Different Dimensions of Sounds

• Length

• Amplitude

• Pitch

• …you name it …

Amplitude of mismatch response can be used as a measure of perceptual distance

Outline


‘Vowel Space’

Näätänen et al. (1997)

e e/ö ö õ o

Place of Articulation

• Formant Transition Cues

[bæ]

[dæ]


Sharma et al. 1993


• Hindi and Salishcontrasts testedon English kids

Janet Werker

U. of British Columbia

6-12 Months: What Changes?

Structure Changing

Patricia KuhlU. of Washington


• Non-native continuumb -- d -- D

• 3 contrastsNative b -- dNon-native d -- DNon-phonetic b1 -- b5

• Conflicting results!





Dehaene-Lambertz 1997





Rivera-Gaxiola et al. 2000





Tsui et al. 2000

Interim Conclusion

• MMN/MMF is a sensitive measure of discrimination

• In some - but not all - cases, MMN amplitude tracks native language discrimination patterns

• When MMN fails to show native language category effects…– could reflect that MMN accesses only low-level acoustic

representations

– could reflect that MMN accesses multiple levels of representation, but response is dominated by acoustic representation

• These studies all implicate phonetic categories

Outline


Objective

• Isolate phonological categories, not phonetic categories

A Category

Another Category

3

III

Auditory Cortex Accesses Phonological Categories: An MEG Mismatch Study

Colin Phillips, Tom Pellathy,Alec Marantz, Elron Yellin, et al.

Journal of Cognitive Neuroscience, 2000

Voice Onset Time (VOT)

60 msec

Categorical Perception

Design

0ms 8ms 16ms 24ms 40ms 48ms 56ms 64ms

20ms 40ms 60ms

Fixed Design - Discrimination

Grouped Design - Categorization

Phonological Features in Auditory Cortex

Colin PhillipsTom PellathyAlec Marantz

Sound Groupings

(Phillips, Pellathy & Marantz 2000)

Phonological Features

Roman Jakobson, 1896-1982

Phonological Natural Classes exist because...

• Phonemes are composed of features - the smallest building blocks of language

• Phonemes that share a feature form a natural class

Effect of Feature-based organization observed in…

• Language development• Language disorders• Historical change• Synchronic processes

Voicing in English

[+voice] [-voice]

Past tense-ed

rub[d]rig[d]

complain[d]

rip[t]lack[t]wish[t]

Plural-s

rub[z]fig[z]

plain[z]

rip[s]rack[s]mat[s]

Japanese - Rendaku

• take + sao takezao

• cito + tsuma citozuma

• hon + tana hondana

• yo: + karasi yo:garasi

• asa + furo asaburo

• Second member of compound word

s zts zt dk gf b

[-voice] [+voice]

Sound Groupings in the Brain

pæ, tæ, tæ, kæ, dæ, pæ, kæ, tæ, pæ, kæ, bæ, tæ...




pæ, tæ, tæ, kæ, dæ, pæ, kæ, tæ, pæ, kæ, bæ, tæ...

Feature Mismatch: Stimuli


Feature MismatchDesign



pæ tæ tæ kæ dæ pæ kæ tæ pæ kæ bæ tæ ...




– – – – [+voi]




– – – – [+voi] – – – – – [+voi] – …




– – – – [+voi] – – – – – [+voi] – …

• Voiceless phonemes are in many-to-one ratio with [+voice] phonemes

• No other many-to-one ratio in this sequence(Phillips, Pellathy & Marantz 2000)

Feature Mismatch


Feature Mismatch

Left Hemisphere Right Hemisphere


Control Experiment - ‘Acoustic Condition’

• Identical acoustical variability• No phonological many-to-one ratio


Feature Mismatch


• Studies of acoustic and phonetic contrasts consistently report bilateral mismatch responses

Paavilainen, Alho, Reinikainen et al. 1991; Näätänen & Alho, 1995; Levänen, Ahonen, Hari et al. 1996; Alho, Winkler, Escera et al. 1998; Ackermann, Lutzenberger & Hertrich, 1999; Opitz, Mecklinger, von Cramon et al. 1999, etc., etc.

• Striking difference in our finding of a left-hemisphere only mismatch response elicited by phonological feature contrast

• Our studies probe a more abstract level of phonological representation

Hemispheric Contrast in MMF

EEG Measures of Discrimination and Categorization of Speech Sound Contrasts

Colin PhillipsShani Abada

Daniel Garcia-PedrosaNina Kazanina

Design

0ms 8ms 16ms 24ms 40ms 48ms 56ms 64ms

20ms 40ms 60ms

Fixed Design - Discrimination

Grouped Design - Categorization

Voice Onset Time (VOT) Mismatch Negativity (MMN): An ERP study

• MMN: acoustic or perceptual phenomenon?• Does an across-category distinction (20ms VOT /da/, 40ms VOT /ta/ evoke a greater MMN than a within-category distinction

(40ms VOT /ta/, 60ms VOT /ta/)?• Sharma & Dorman (1999): MMN only across categories; MMN represents perceptual, not physical, difference between

stimuli; Double N100 for long VOT• ‘Oddball’ paradigm - 7:1 ratio of standards to deviants

Fixed conditionstandard VOT = 20ms, deviant VOT = 40 ms (across), or standard VOT = 40ms, deviant

VOT = 60ms (within)

Grouped conditionno specific standard VOT,

but 7/8 fall into either /da/ or /ta/

20st,40dv 40st,60dv Dst,Tdv Tst,Ddv

Discrimination and Categorization of Vowels and Tones

Daniel Garcia-PedrosaColin Phillips

Two Concerns

• Are the category effects an artifact:– it is very hard to discriminate different members of the

same category on a voicing scale

– subjects are forming ad hoc groupings of sounds during the experiment, and are not using their phonological representations?

Discrimination





0ms

20ms

40ms

20ms

40ms

60ms

D T

D

T T

D


Vowels

• Vowels show categorical perception effects in identification tasks

• …but vowels show much better discriminability of within-category pairs

Method: Materials

• Tones: 290Hz, 300Hz, 310 Hz…470Hz

• Vowels– First formant (F1) varies along the same 290-

470Hz continuum– F0, F2, voicing onset, etc. all remain constant

Method: Procedure

• Subject’s category boundary determined by pretest

• Grouped mismatch paradigm– Standard stimulus (7/8) = 4 exemplars from one category

– Deviant stimulus (1/8) = 4 exemplars from other category

– MMN response therefore = deviance from a category, not from a single stimulus

• Tones and vowels presented in separate blocks

Results: Vowels

Results: Vowels

Results: Tones

Results: Tones

Preliminary conclusions

• MMN appears about 150ms post-stimulus in vowel but not in tone condition

• Higher amplitude N100 for deviants in both conditions. Is this evidence for categorization of tones or just the result of habituation?

• Acoustic differences may be responsible for greater N100, while categorization elicits the MMN

Phonemic vs. Allophonic Contrasts

Nina KazaninaColin Phillips

in progress


• Focus on meaning-relevant sound contrasts

Russian d t

Korean d t



Russian d t

Korean d t



Russian d t

Korean d t

…ada ada ada ada ada ada ata ada ada ada ata…

EXECTIVE SUITE

• Japanese versus French

• Pairs like “egma” and “eguma”

• Difference is possible in French, but not in Japanese

Phonology - Syllables

Behavioral Results

• Japanese have difficulty hearing the difference

Dupoux et al. 1999

ERP Results

• Sequences: egma, egma, egma, egma, eguma

• French have 3 mismatch responses– Early, middle, late

• Japanese only have late response

Dehaene-Lambertz et al. 2000

ERP Results - 2• Early response


ERP Results - 3

• Middle response


ERP Results - 4

• Late response



Thai speakers:

Thai *words*: [da] [ta] DIFFERENT

English *words*: [daz] [taz] SAME

Imsri (2001)

• Voiceless stops /p, t, k/

• Aspirated at start of syllable; unaspirated after [s]

pitspitspitbit

tackstackstackdack

Varying Pronunciations

Outline


Conclusion

• Sound representations involve (multiple degrees of) abstraction

• Different levels of representation develop independently from 0-18 months of age

• Although much is known about course of development, many open questions about how change proceeds

• Possibility of a connection between adult electrophysiology and infant developmental findings

Documents

Speech Perception in Infant and Adult Brains Colin Phillips Cognitive Neuroscience of Language Laboratory Department of Linguistics University of Maryland