Artikler til reeksamen, Kognition 2018

1

Eksamensartikler til reeksamen (feb. 2019) BA Kognitionspsykologi Teori og Metode Efterår 2018

1. Sansning og perception Susilo, T., Yovel, G., Barton, J. J. S., Duchaine, B. (2013). Face perception is category-specific: Evidence from normal body perception in acquired prosopagnosia. Cognition, 129(1), 88-94. (6 ns)

2. Opmærksomhed Marois, R., Yi, D. J., & Chun, M. M. (2004). The neural fate of consciously perceived and missed events in the attentional blink. Neuron, 41(3), 465-472. (9,5 ns)

3. Korttidshukommelse/Arbejdshukommelse Vogel, E. K., & Machizawa, M. G. (2004). Neural activity predicts individual differences in visual working memory capacity. Nature, 428(6984), 748-751. (4 ns)

4. Indlæring og langtidshukommelse Chang, Q., & Gold, P. E. (2003). Switching memory systems during learning: changes in patterns of brain acetylcholine release in the hippocampus and striatum in rats. Journal of Neuroscience, 23(7), 3001-3005. (9,5 ns)

5. Semantik, kategorisering og mentale repræsentationer Dehaene, S., Naccache, L., le Clec’H, G., Koechlin, E., Mueller, M., Dehaene-Lambertz, G. et al. (1998). Imaging unconscious semantic priming. Nature, 395, 597-600. (4 ns)

6. Sprog Tanenhaus, M. K., Spivey-Knowlton, M. J., Eberhard, K. M., & Sedivy, J. C. (1995). Integration of visual and linguistic information in spoken language comprehension. Science, 1632-1634. (4 ns)

7. Emotioner og social kognition Schneider, D., Lam, R., Bayliss, A. P., & Dux, P. E. (2012). Cognitive load disrupts implicit theory-of-mind processing. Psychological Science, 23, 842-847. (5 ns)

8. Beslutningstagning, tænkning og problemløsning Kounios, J., Frymiare, J. L., Bowden, E. M., Fleck, J. I., Subramaniam, K., Parrish, T. B., & Jung-Beeman, M. (2006). The prepared mind neural activity prior to problem presentation predicts subsequent solution by sudden insight. Psychological Science, 17(10), 882-890. (8 ns) 9. Eksekutive funktioner og kognitiv kontrol Janssen, T. W., Heslenfeld, D. J., van Mourik, R., Logan, G. D., & Oosterlaan, J. (2015). Neural correlates of response inhibition in children with attention-deficit/hyperactivity disorder: a controlled version of the stop-signal task. Psychiatry Research: Neuroimaging, 233(2), 278-284. (10 ns)

Artikler til reeksamen, Kognition 2018

2

Eksamensdiskussionen relateret til hver artikel vil primært bevæge sig inden for følgende teoretiske hovedemner. Emnets andel af pensum afspejles i sandsynligheden for at trække hver artikel (”vægt”). Desuden vil den anvendte metode i artiklen (eksempelvis reaktionstids-eksperiment, EEG eksperiment, fMRI scanning, patientstudium, dyremodel) blive inddraget i diskussionen.

1. Sansning og perception (vægt: 2/17) 2. Opmærksomhed (vægt: 2/17) 3. Korttidshukommelse/Arbejdshukommelse (vægt: 1/17) 4. Indlæring og langtidshukommelse (vægt: 4/17) 5. Semantik, kategorisering og mentale repræsentationer (vægt: 2/17) 6. Sprog (vægt: 2/17) 7. Emotioner og social kognition (vægt: 1/17) 8. Beslutningstagning, tænkning og problemløsning (vægt: 2/17) 9. Eksekutive funktioner og kognitiv kontrol (vægt: 1/17)

Cognition 129 (2013) 88–94

Contents lists available at SciVerse ScienceDirect

Cognition

journal homepage: www.elsevier .com/ locate/COGNIT

Brief article

Face perception is category-specific: Evidence from normalbody perception in acquired prosopagnosia

0010-0277/$ - see front matter � 2013 Elsevier B.V. All rights reserved.http://dx.doi.org/10.1016/j.cognition.2013.06.004

⇑ Corresponding author.E-mail address: bagus.t.susilo@dartmouth.edu (T. Susilo).

Tirta Susilo a,⇑, Galit Yovel b, Jason J.S. Barton c, Bradley Duchaine a

a Department of Psychological and Brain Sciences, Dartmouth College, United Statesb School of Psychological Sciences & Sagol School of Neuroscience, Tel Aviv University, Israelc Departments of Medicine (Neurology) & Opthalmology and Visual Sciences, University of British Columbia, Canada

a r t i c l e i n f o a b s t r a c t

Article history:Received 16 November 2012Revised 10 June 2013Accepted 10 June 2013

Keywords:FaceBodyPerceptionProsopagnosiaInversionExpertise

Does the human visual system contain perceptual mechanisms specialized for particularobject categories such as faces? This question lies at the heart of a long-running debatein face perception. The face-specific hypothesis posits that face perception relies on mech-anisms dedicated to faces, while the expertise hypothesis proposes that faces are processedby more generic mechanisms that operate on objects we have extended experience with.Previous studies that have addressed this question using acquired prosopagnosia areinconclusive because the non-face categories tested (e.g., cars) were not well-matched tofaces in terms of visual exposure and perceptual experience. Here we compare perceptionof faces and bodies in four acquired prosopagnosics. Critically, we used face and body tasksthat generate comparable inversion effects in controls, which indicates that our tasksengage orientation-specific perceptual mechanisms for faces and bodies to a similar extent.Three prosopagnosics were able to discriminate bodies normally despite their impairmentin face perception. Moreover, they exhibited normal inversion effects for bodies, suggestingtheir body perception was carried out by the same mechanisms used by controls. Our find-ings indicate that the human visual system contains processes specialized for faces.

� 2013 Elsevier B.V. All rights reserved.

1. Introduction

A fundamental issue in cognitive science concerns theextent to which the human mind consists of processes spe-cialized for particular object categories. This issue moti-vates the long-running debate about the nature of faceperception. According to the face-specific hypothesis, faceperception is carried out by mechanisms specialized forfaces (Pitcher, Charles, Devlin, Walsh, & Duchaine, 2009;Tanaka & Farah, 1993; Yin, 1969). According to the exper-tise hypothesis, faces are analyzed by more generic mech-anisms for objects with which we have extendedexperience (Diamond & Carey, 1986; Gauthier & Tarr,1997; McGugin, Gatenby, Gore, & Gauthier, 2012). Here

we contrast the two hypotheses by examining body per-ception when face perception is impaired in acquired pros-opagnosia (Bodamer, 1947).

Previous studies that have investigated the nature offace processing using acquired prosopagnosia have typi-cally compared perception of faces and a variety of non-face objects (e.g., Busigny, Graf, Mayer, & Rossion, 2010;Farah, Klein, & Levinson, 1995; Moscovitch, Winocur, &Behrmann, 1997; Riddoch, Johnston, Bracewell, Boutsen,& Humphreys, 2008). While dissociations between percep-tion of faces and non-faces suggest that faces are processeddifferently than most objects, they do not distinguish be-tween the face-specific hypothesis and the expertisehypothesis because (i) both hypotheses agree that facesare processed by mechanisms different from those usedfor objects (i.e., most of us are experts with faces but notobjects), and (ii) the non-face categories tested (e.g., cars,chairs) are not matched to faces in terms of perceptual

Table 1Face recognition ability (z-scores) of the acquired prosopagnosics. In theCambridge Face Memory Test (CFMT, Duchaine & Nakayama, 2006),participants study six target faces and then select which of three presentedfaces is a target face. In the Cambridge Face Perception Test (CFPT,Duchaine, Yovel, & Nakayama, 2007), participants sort six faces based totheir similarity to a target face simultaneously presented in a differentview. In the Queen Square Face Identity Test (Garrido et al., 2009),participants make a same/different identity judgment on two sequentiallypresented faces that always differ in expression. All z-scores are more thantwo standard deviations below the mean (except Galen’s z-score on theCFPT), indicating severe impairment in face processing. All z-scores werecomputed using control means in the cited references.

Florence Sandy Grace Galen

Cambridge Face Memory Test �4.66 �4.29 �3.53 �3.78Cambridge Face Perception

Test�3.65 �3.38 �3.24 �1.26

Queen Square Identity Test �4.33 �2.77 �2.31 �2.33

T. Susilo et al. / Cognition 129 (2013) 88–94 89

experience. To discriminate between the two hypotheses,faces need to be compared with an object category forwhich participants have similar amounts of perceptualexperience. Only then will the expertise hypothesis predictan association between faces and non-faces in all prosop-agnosics, while the face-specific hypothesis suggest disso-ciations can occur in some prosopagnosics.

Here we used bodies as a comparison category, becausefaces and bodies share two theoretically important charac-teristics. First, both faces and bodies produce inversion ef-fects (i.e. worse discrimination of visual stimuli presentedupside-down) larger than those for other objects (the faceinversion effect, Yin, 1969; the body inversion effect, Reed,Stone, Bozova, & Tanaka, 2003). This is important becauseinversion effects indicate orientation-specific processingand are considered a marker of perceptual expertise. Mostcritical for our study, inversion effects for faces and bodiescan be similar in size (Robbins & Coltheart, 2012a; Yovel,Pelc, & Lubetzky, 2010), which indicates that faces andbodies can engage orientation-specific mechanisms to asimilar extent. Second, faces and bodies exhibit consistentfirst-order configurations (i.e., fixed spatial relations be-tween eyes, nose, and mouth for faces; arms, torso, andlegs for bodies), which have been suggested to be neces-sary for the development of visual expertise for particularobject categories (Diamond & Carey, 1986).

Our study consisted of three steps. We first confirmedthat our face and body tasks generate comparable inver-sion effects in controls. This step ensured that our tasks en-gage orientation-specific processing of faces and bodies toa similar extent, a critical factor in contrasting the face-specific and the expertise hypotheses. Next we comparedhow the prosopagnosics discriminate among exemplarsof upright faces and of upright bodies. Finally we examinedwhether the prosopagnosics who were able to discriminateupright bodies as accurately as controls also showed nor-mal-sized inversion effects for bodies, which would sug-gest that their body perception was generated by thesame mechanisms used by controls. The status of the bodyinversion effect in acquired prosopagnosia is of additionalinterest because there is some evidence that the bodyinversion effect might involve face-selective rather thanbody-selective neural mechanisms (Brandman & Yovel,2010).

2. Method

2.1. Participants

We tested four acquired prosopagnosics, namely Flor-ence, Sandy, Grace, and Galen, as part of our broader inves-tigation of prosopagnosia. Table 1 shows their performanceon tests of face recognition.

Florence is a right-handed nurse born in 1982. She was29 years old when tested. In 2006, Florence noticed prob-lems with face recognition following a right amygdalo-hip-pocampectomy. Functional MRI scans showed bilateralactivations in her fusiform face area, occipital face area,and superior temporal sulcus. In 2008 she underwent asecond surgery that removed the anterior third of her right

temporal lobe, sparing the core face areas previously iden-tified. Florence did not complain of visual impairmentsother than prosopagnosia, and she performed normallyon within-class recognition of objects including hairstyles,cars, and abstract paintings. In Fox, Hanif, Iaria, Duchaine,and Barton (2011), Florence was referred to as R-AT1.

Sandy is a right-handed woman born in 1975. She was36 years old when tested. Sandy became prosopagnosicafter a right hippocampal resection in 2003 during whichshe had a stroke in the occipital lobe and she lost her leftvisual field completely. She complained of severe difficul-ties recognizing faces, including herself in the mirror andher children in the school, and reported that she reliesheavily on gait and walking sound to identify people. San-dy also complained of object recognition problems, such asfinding her car when other cars in the parking lot have sim-ilar colors. Sandy was impaired on tests of visual closure,eye gaze perception, facial expression recognition, andhairstyle recognition. Sandy reported no general memoryproblems.

Grace is a right-handed pharmacist born in 1968. Shewas 43 years old when tested. Grace acquired prosopagno-sia after a brain biopsy of the right temporal lobe in 1982to treat herpes simplex viral encephalitis. Grace com-plained about difficulties in face recognition and relies onnon-face cues like voice, hairstyle, glasses, and gait to iden-tify people. In addition to her prosopagnosia, Grace wasalso impaired on tests of color perception, visual closure,and basic object recognition from line drawings. StructuralMRI scans showed a lesion in the right anterior temporallobe extending to the middle fusiform and inferior tempo-ral gyri, as well as a small lesion in the middle aspect of theleft fusiform gyrus. She was referred to as B-OT/AT1 in Dal-rymple et al. (2011).

Galen is a right-handed physician born in 1982. He was29 years old when tested. Galen became prosopagnosic in2004 following a craniotomy for an arteriovenous malfor-mation in his right temporal lobe. He complained of diffi-culties recognizing faces, including celebrities and peoplewho are related or have similar appearances and reportedusing contextual cues to identify people. Galen previouslyexperienced a left-superior quadrantanopia, but a recentexamination showed his low-level abilities in the

Fig. 1. Experimental task showing example stimuli for faces, faceless bodies, and headless bodies.

90 T. Susilo et al. / Cognition 129 (2013) 88–94

left-superior visual field are in the normal range. Galen didnot complain of visual agnosia in general, and he scorednormally on recognition tests involving hairstyles, cars,and abstract paintings. Functional scans showed an ab-sence of right fusiform face area and right occipital facearea.

Twenty people from the Dartmouth College community(13 female, age range 18–27 years) participated ascontrols.

2.2. Stimuli and procedure

The main experiment used a task developed by Yovelet al. (2010), in which participants made same/differentjudgments on 144 sequentially presented pairs of headlessbodies, faceless bodies, and faces, shown in different blocks(Fig. 1). Body pairs differed in terms of, the position of thearms, legs, and heads (in the faceless bodies). Face pairsdiffered in terms of eyes, nose, and mouth. For each ofthe three categories, upright and inverted trials (72 each)were interleaved in a pre-determined random order. Head-less bodies were tested first, faceless bodies second, andfaces last to ensure that poor face discrimination was notdue to unfamiliarity with the paradigm and that normalbody discrimination was not confounded by practice ef-fects. Dependent measures were d-prime and responsetime. Inversion effects were computed as [upright d-pri-me � inverted d-prime] and as [upright RT � invertedRT]. (Note that we also computed inversion effects in a rel-ative manner: [(upright d0 � inverted d)/(uprightd0 + inverted d0)] and [(upright RT � inverted RT)/(uprightRT + inverted RT)]; as presented in Supplementary Figure,we found similar results for all prosopagnosics and thuscame to the same conclusion.)

1 The inversion effect for faceless bodies was trending smaller than thatfor faces (p = 0.09), but two previous studies using the same task found nosuch trend (p = 0.54 in Brandman & Yovel, 2012; p > 0.3 in Yovel et al.,2010). Based on all available data we would argue that our task generatesstatistically comparable inversion effects for faces and faceless bodies.

2.3. Statistical analysis

We used the t-test for single-case analysis (Crawford &Howell, 1998) to compare a case score against the controlmean in a particular condition (e.g., Florence’s discrimina-tion of upright faces). To compare each case’s differencescores (e.g., the difference between Florence’s discrimina-tion of faces and her discrimination of faceless bodies)against the difference scores in controls, we used the

Bayesian Standardized Difference Test (Crawford & Gart-hwaite, 2007). For all statistical analyses we report theestimated percentage of the control population that wouldperform worse than a case score or would exhibit a largerdifference score in the predicted direction. Note that theseestimated percentages directly correspond to p-values.Scores below a 5% cut-off were classified as abnormal.

3. Results

3.1. Did faces and bodies show comparable inversion effects incontrols?

Fig. 2A shows that all conditions produced inversion ef-fects in controls. Computed using the absolute index (i.e.[upright d0 � inverted d0]), the inversion effect for faces(M = 1.28, SE = 0.21) was comparable to that for facelessbodies (M = 1.04, SE = 0.18), F(1,19) = 3.19, p = 0.09, butlarger than that for headless bodies (M = 0.55, SE = 0.18),F(1,19) = 24.91, p < 0.001.1 Computed using the relative in-dex (i.e., [(upright d0 � inverted d0)/(upright d0 + invertedd0)], the inversion effect for faces (M = 0.31, SE = 0.04) wasagain comparable to that for faceless bodies (M = 0.26,SE = 0.03), F(1,19) = 1.52, p = 0.23, and larger than that forheadless bodies (M = 0.16, SE = 0.07), F(1,19) = 4.9, p = 0.04.Fig. 2B shows that there was no speed–accuracy trade-off:participants were slower to discriminate inverted than up-right stimuli. This result replicates a previous finding (Yovelet al., 2010), and indicates that faces and faceless bodies, butnot headless bodies, engage orientation-specific processes toa similar degree.

3.2. How did the prosopagnosics discriminate faces and bodiesin the upright orientation?

Table 2 (condition scores) shows that all prosopagno-sics, except Sandy, were statistically impaired with facesbut normal with faceless bodies and headless bodies onboth d-prime and RT. However, given that a statistically

Fig. 2. Inversion effects in controls for faces, faceless bodies, and headless bodies in terms of (A) d0 and (B) response time.

Table 2Raw scores of the prosopagnosics and associated estimates of % of the control population that would perform worse than each score. Abnormal performancesare indicated in italics.

Florence Sandy Grace Galen Control M Control SD Florence Sandy Grace Galen

d-PrimeCondition scores (upright)

Faces 0.54 0.43 �0.42 1.02 2.74 0.90 1.38 1.08 0.14 3.88Faceless bodies 1.66 1.02 1.50 1.90 2.37 0.77 18.97 5.17 14.20 27.92Headless bodies 1.25 1.17 0.46 1.48 1.62 0.79 32.64 29.24 8.40 43.23

Difference scores (upright)Faces–faceless bodies �1.12 �0.59 �1.92 �0.88 0.37 0.57 2.12 13.83 0.20 3.61Faces–headless bodies �0.71 �0.74 �0.88 �0.46 1.12 0.62 0.89 0.87 1.10 1.60

Inversion effectsFaces 0.38 �0.20 �0.49 0.80 1.28 0.72 37.77 11.43 12.56 45.14Faceless bodies 0.79 0.15 0.64 0.57 1.04 0.68 38.10 31.79 44.77 27.84Headless bodies �0.34 0.30 �0.26 �0.64 0.55 0.84 9.76 48.31 34.00 2.44

Response time (ms)Condition scores (upright)

Faces 2315 1948 822 1352 955 236 0.00 0.00 70.58 5.46Faceless bodies 1083 1953 1159 736 994 263 37.22 0.01 27.40 82.49Headless bodies 1297 1874 1143 769 1132 276 28.35 0.01 48.47 89.26

Difference scores (upright)Faces – faceless bodies 1232 �5 �337 616 �39 187 0.01 10.51 26.80 0.10Faces – headless bodies 1018 74 �322 584 �176 250 0.00 28.20 6.29 0.29

Inversion effectsFaces 134 276 �231 85 �159 154 0.03 0.01 20.62 0.76Faceless bodies �129 76 18 �56 �106 145 47.37 3.86 16.39 48.28Headless bodies �179 �173 �15 �91 �48 134 24.13 46.82 40.09 22.60

T. Susilo et al. / Cognition 129 (2013) 88–94 91

impaired score and a non-impaired score may not be sig-nificantly different (Crawford & Garthwaite, 2007), we nextassessed whether the differences between discrimination offaces and bodies in the prosopagnosics were statisticallyabnormal compared to the same differences in controls.

3.3. Was the difference between discrimination of faces andfaceless bodies abnormal?

The difference scores in Table 2 show that the d-primesfor Florence, Grace, and Galen were significantly worse forfaces than for faceless bodies. Florence and Galen were alsosignificantly different on RT. Critically, all three prosopag-

nosics exhibited normal-sized inversion effects for facelessbodies on both d-prime and RT (Table 2 inversion effects,Fig. 3). Normal discrimination of and normal-sized inver-sion effects for faceless bodies indicates that despite theirprosopagnosia, Florence, Grace, and Galen processed face-less bodies like controls did.

3.4. Was the difference between discrimination of faces andheadless bodies abnormal?

The difference scores in Table 2 also show that all pros-opagnosics performed worse with faces than with headlessbodies on d-prime. As above, Florence and Galen also

Fig. 3. Individual inversion effects (d-prime and response time) for Florence (red), Sandy (blue), Grace (green), and Galen (yellow) relative to individualcontrols (grey) for (A) faces, (B) faceless bodies, and (C) headless bodies. See Supplementary Figure for similar plots of inversion effects computed using therelative index. (For interpretation of the references to color in this figure legend, the reader is referred to the web version of this article.)

92 T. Susilo et al. / Cognition 129 (2013) 88–94

showed a dissociation for RT. Florence, Sandy, and Graceshowed normal-sized inversion effects for headless bodieson both d-prime and RT (Table 2 inversion effects, Fig. 3C).These results indicate that perception of headless bodiescan be spared in prosopagnosia, although it does not dis-tinguish between the face-specific and the expertisehypotheses because the inversion effect for headlessbodies in controls was smaller than that for faces to startwith.

3.5. Did the prosopagnosics who show normal body inversioneffects also show normal face inversion effects?

The idea that body inversion effects might rely onmechanisms for face rather than body perception (Brand-man & Yovel, 2010) predicts that prosopagnosics whoshowed normal body inversion effects should also shownormal face inversion effects. Is this the case? Our datasuggest not. While Florence, Grace, and Galen showed faceinversion effects in the normal range ond-prime, Florence

and Galen exhibited a clear speed/accuracy trade-off: theywere much slower with upright faces than with invertedfaces (Table 2 inversion effects, Fig. 3A). As a result, theirdata are difficult to interpret. In contrast, Florence and Ga-len showed normal-sized inversion effects for facelessbodies on both d-prime and RT, thus indicating a dissocia-tion between their inversion effects for faces and for face-less bodies.

3.6. Was discrimination of bodies easier than discrimination offaces?

None of our results can be accounted by easier discrim-ination of bodies than of faces. In fact controls were betterat discriminating faces than both faceless bodies,t(19) = 2.90, p < .01, and headless bodies, t(19) = 8.08,p < .0001. This means that three of four prosopagnosicsperformed normally on body tasks that are more challeng-ing than the face task they had impairments with.

T. Susilo et al. / Cognition 129 (2013) 88–94 93

4. Discussion

In this study we addressed whether faces are processedby specialized mechanisms (face-specific hypothesis) or bymore generic mechanisms for objects we have extendedexperience with (expertise hypothesis). We did so by com-paring perception of faces and bodies in acquired proso-pagnosia, because faces and bodies engage orientation-specific perceptual mechanisms and exhibit consistentfirst-order configurations among their parts. Controlsexhibited comparable inversion effects for faces and face-less bodies. Three of four prosopagnosics were able to dis-criminate bodies as well as controls and showed normal-sized body inversion effects. Their results indicate bodyperception can be normal when face perception is im-paired, consistent with the face-specific hypothesis.

Our findings add to the literature on acquired prosop-agnosics who performed normally with non-faces.Termed ‘‘pure’’ prosopagnosics (for a review of 13 exist-ing cases see Busigny et al., 2010), these participants areoften considered evidence that faces are processed bydedicated mechanisms, especially when the face andnon-face tasks are matched for task demands, equatedin difficulty, and free of speed/accuracy trade-offs. How-ever, such dissociations do not discriminate between theface-specific and the expertise hypotheses because it isunclear whether the prosopagnosics had extensive expe-rience with the non-face categories tested. In contrast,our use of bodies as a comparison category allows usto tease apart the two hypotheses because faces andbodies share theoretically-important characteristics men-tioned above.

Our study is the first to report a dissociation betweenface and body perception in acquired prosopagnosia. CaseFM was impaired with perception of both faces and bodies(Moro et al., 2012). The well-known case PS showed typicalfMRI activation in body-selective areas to emotional bodystimuli, but her behavioral performance with bodies wasnot assessed (Peelen, Lucas, Mayer, & Vuilleumier, 2009).A few studies examined face and body perception in devel-opmental prosopagnosia and found mixed results (e.g.,Duchaine, Yovel, Butterworth, & Nakayama, 2006; Righart& De Gelder, 2007). Crucially, however, none of these stud-ies demonstrated that the body tasks used generated face-size inversion effects in controls. It thus remains possiblethat normal body perception observed in developmentalprosopagnosics did not depend on orientation-specificmechanisms to a similar extent as faces did.

Our result agrees with evidence from single-cell, func-tional imaging (fMRI), and transcranial magnetic stimula-tion (TMS) studies. The existence of face-selectiveneurons has long been reported (Gross, Rocha-Miranda, &Bender, 1972; Perrett, Rolls, & Caan, 1982), and recentinvestigations have established that these neurons arefunctionally organized in a network of face-selectivepatches (Moeller, Freiwald, & Tsao, 2008). Functional imag-ing studies have found separate cortical areas selective forfaces and bodies in humans (de Gelder et al., 2010; Peelen& Downing, 2007), and TMS studies have indicated thecausal involvement of some of these areas in discrimina-tion tasks only for their preferred category (Pitcher et al.,

2009; Urgesi, Berlucchi, & Aglioti, 2004). Our finding com-plements these data by showing a cognitive dissociationbetween face and body perception. While direct mappingbetween behavioral performance and lesion location is be-yond the scope of the present study, future studies of ac-quired prosopagnosia are likely to benefit from obtainingfunctional scans to body and body part stimuli.

What is the nature of the orientation-specific mecha-nisms for faceless bodies that were spared in our prosopag-nosics? Given their face-size inversion effects, thesemechanisms might perform holistic computations similarto those in face perception. Consistent with this possibility,perception of body parts benefits from the presence of thewhole body (Seitz, 2002), just like perception of face partsbenefits from the context of the whole face (i.e., the part-whole effect, Tanaka & Farah, 1993). Perception of one-halfof bodies can be influenced by the unattended half (Rob-bins & Coltheart, 2012b, but see Soria Bauser, Suchan, &Daum, 2011), similar to the composite face effect (Young,Hellawell, & Hay, 1987). Future studies can use these par-adigms to further clarify whether normal body perceptionin acquired prosopagnosia is holistic in nature.

A recent study however suggests that the face-sizeinversion effect for faceless bodies might not be drivenby body perception mechanisms, but instead by face detec-tion mechanisms (Brandman & Yovel, 2012). This studycompared the size of inversion effects for several body con-ditions including faceless bodies, heads with shoulders,heads only, and bodies from the back. Only faceless bodiesand heads with shoulders generated face-size inversion ef-fects; other conditions produced smaller inversion effects.In a second experiment, different body conditions wereflashed for 17 ms each, and participants were askedwhether they saw a face. Interestingly, participants weremore likely to report seeing a face in the same two condi-tions that produced face-size inversion effects, namelyfaceless bodies and heads with shoulders. The authorsinterpreted these results as evidence that the face-sizeinversion effects in bodies are generated by face detectionmechanisms.

Although our data are not inconsistent with the poten-tial involvement of face detection mechanisms (we did notsystematically assess face detection ability of the prosop-agnosics), it is worth noting that face-size inversion effectswere obtained for faceless bodies and bodies with shoul-ders, not for heads only (Brandman & Yovel, 2012). Thismeans two things: (i) there has to be some body parts inthe stimuli (shoulders at the minimum) for the face-sizeinversion effects to emerge, and (ii) these body parts haveto be processed normally. A participant with impairedshoulder perception, for example, would be expected toprocess faceless bodies abnormally, and thus would failto exhibit normal inversion effects. The fact that our pros-opagnosics exhibited normal inversion effects for facelessbodies implies that their ability to process all aspects offaceless bodies was normal.

Regardless of the underlying mechanisms, the bodyinversion effect indicates that bodies, unlike most non-faceobjects, are processed by perceptual mechanisms that arevery sensitive to orientation and are therefore a suitablecategory for distinguishing between the expertise and the

94 T. Susilo et al. / Cognition 129 (2013) 88–94

face-specific hypotheses. No prosopagnosia studies to datehave used a task where the non-face category is compara-ble to faces in terms of sensitivity to orientation, and thusour study offers a critical piece of evidence that is inconsis-tent with the notion that prosopagnosia is an impairmentaffecting the processing of objects with which we have ex-tended experience. Rather, our findingssuggest prosopag-nosia can be a category-specific deficit that is restrictedto faces, which indicates the human mind contains pro-cesses specialized for particular object categories.

Acknowledgements

We thank Florence, Sandy, Grace, and Galen for theirparticipation, and Constantin Rezlescu for testing Sandyand Grace. This work was supported in part by CIHR GrantMOP-102567 and the Hitchcock Foundation. J.J.S.B. wassupported by a Canada Research Chair.

Appendix A. Supplementary material

Supplementary data associated with this article can befound, in the online version, at http://dx.doi.org/10.1016/j.cognition.2013.06.004.

References

Bodamer, J. (1947). Die Prosop-Agnosie. Archiv für Psychiatrie undNervenkrankheiten, 179, 6–53.

Brandman, T., & Yovel, G. (2012). TA face inversion effect without a face.Cognition, 125, 365–372.

Brandman, T., & Yovel, G. (2010). The body inversion effect is mediated byface-selective not body-selective areas. Journal of Neuroscience, 30,10534–10540.

Busigny, T., Graf, M., Mayer, E., & Rossion, B. (2010). Acquiredprosopagnosia as a face-specific disorder: Ruling out the generalvisual similarity account. Neuropsychologia, 48, 2051–2067.

Crawford, J. R., & Garthwaite, P. H. (2007). Comparison of a single case to acontrol or normative sample in neuropsychology: Development of aBayesian approach. Cognitive Neuropsychology, 24, 343–372.

Crawford, J. R., & Howell, D. C. (1998). Comparing an individual’s testscore against norms derived from small samples. The ClinicalNeuropsychologist, 12, 482–486.

Dalrymple, K. A., Oruç, I., Duchaine, B., Pancaroglu, R., Fox, C. J., Iaria, G.,et al. (2011). The neuroanatomic basis of the right face-selective N170in acquired prosopagnosia: A combined ERP/fMRI study.Neuropsychologia, 49, 2553–2563.

de Gelder, B., Van den Stock, J., Meeren, H. K. M., Sinke, C. B. A., Kret, M. E.,& Tamietto, M. (2010). Standing up for the body. Recent progress inuncovering the networks involved in processing bodies and bodilyexpressions. Neuroscience and Biobehavioral Reviews, 34, 513–527.

Diamond, R., & Carey, S. (1986). Why faces are and are not special: Aneffect of expertise. Journal of Experimental Psychology: General, 115,107–117.

Duchaine, B., & Nakayama, K. (2006). The Cambridge Face Memory Test:Results for neurologically intact individuals and an investigation of itsvalidity using inverted face stimuli and prosopagnosic patients.Neuropsychologia, 44, 576–585.

Duchaine, B., Yovel, G., Butterworth, E., & Nakayama, K. (2006).Prosopagnosia as an impairment to face-specific mechanisms:Elimination of the alternative hypotheses in a developmental case.Cognitive Neuropsychology, 23, 714–747.

Duchaine, B., Yovel, G., & Nakayama, K. (2007). No global processingdeficit in the Navon task in 14 developmental prosopagnosics. SocialCognitive and Affective Neuroscience, 2, 104–113.

Farah, M. J., Klein, K. L., & Levinson, K. L. (1995). Face perception andwithin-category discrimination in prosopagnosia. Neuropsychologia,33, 661–674.

Fox, C. J., Hanif, H. M., Iaria, G., Duchaine, B., & Barton, J. J. S. (2011).Perceptual and anatomic patterns of selective deficits in facialidentity and expression processing. Neuropsychologia, 49, 3188–3200.

Garrido, L., Furl, N., Draganski, B., Weiskopf, N., Stevens, J., Tan, G. C.-Y.,et al. (2009). VBM reveals reduced gray matter volume in thetemporal cortex of developmental prosopagnosics. Brain, 132,3443–3455.

Gauthier, I., & Tarr, M. J. (1997). Becoming a ‘‘Greeble’’ expert: Exploringmechanisms for face recognition. Vision Research, 37, 1673–1682.

Gross, C. G., Rocha-Miranda, C. E., & Bender, D. B. (1972). Visual propertiesof neurons in inferotemporal cortex of the macaque. Journal ofNeurophysiology, 35, 96–111.

McGugin, R. W., Gatenby, J. C., Gore, J. C., & Gauthier, I. (2012). High-resolution imaging of expertise reveals reliable object selectivity inthe FFA related to perceptual performance. Proceedings of the NationalAcademy of Sciences of the United States of America, 109, 17063–17068.

Moeller, S., Freiwald, W. A., & Tsao, D. Y. (2008). Patches with links: Aunified system for processing faces in the macaque temporal lobe.Science, 320, 1355–1359.

Moro, V., Pernigo, S., Avesani, R., Bulgarelli, C., Urgesi, C., Candidi, M., et al.(2012). Visual body recognition in a prosopagnosic patient.Neuropsychologia, 50, 104–117.

Moscovitch, M., Winocur, G., & Behrmann, M. (1997). What is specialabout face recognition? Nineteen experiments on a person withvisual object agnosia and dyslexia but normal face recognition.Journal of Cognitive Neuroscience, 9, 555–604.

Peelen, M. V., & Downing, P. E. (2007). The neural basis of visual bodyperception. Nature Reviews Neuroscience, 8, 636–648.

Peelen, M. V., Lucas, N., Mayer, E., & Vuilleumier, P. (2009). Emotionalattention in acquired prosopagnosia. Social Cognitive and AffectiveNeuroscience, 4, 268–277.

Perrett, D. I., Rolls, E. T., & Caan, W. (1982). Visual neurons responsive tofaces in the monkey temporal cortex. Experimental Brain Research, 47,329–342.

Pitcher, D., Charles, L., Devlin, J., Walsh, V., & Duchaine, B. (2009). Tripledissociation between faces, bodies, and objects in extrastriate cortex.Current Biology, 19, 319–324.

Reed, C. L., Stone, V. E., Bozova, S., & Tanaka, J. (2003). The body-inversioneffect. Psychological Science, 14, 302–308.

Riddoch, M. J., Johnston, R. A., Bracewell, R. M., Boutsen, L., & Humphreys,G. W. (2008). Are faces special? A case of pure prosopagnosia.Cognitive Neuropsychology, 25, 3–26.

Righart, R., & de Gelder, B. (2007). Impaired face and body perception indevelopmental prosopagnosia. Proceedings of the National Academy ofSciences of the United States of America, 104, 17234–17238.

Robbins, R. A., & Coltheart, M. (2012a). The effects of inversion andfamiliarity on face versus body cues to person recognition. Journal ofExperimental Psychology: Human Perception and Performance, 38,1098–1104.

Robbins, R. A., & Coltheart, M. (2012b). Left–right holistic integration ofhuman bodies. Quarterly Journal of Experimental Psychology, 65,1962–1974.

Seitz, K. (2002). Parts and wholes in person recognition: Developmentaltrends. Journal of Experimental Child Psychology, 82, 367–381.

Soria Bauser, D. A., Suchan, B., & Daum, I. (2011). Differences betweenperception of human faces and body shapes: Evidence from thecomposite illusion. Vision Research, 51, 195–202.

Tanaka, J. W., & Farah, M. J. (1993). Parts and wholes in face recognition.Quarterly Journal of Experimental Psychology A, 46, 225–245.

Urgesi, C., Berlucchi, G., & Aglioti, S. M. (2004). Magnetic stimulation ofextrastriate body area impairs visual processing of nonfacial bodyparts. Current Biology, 14, 2130–2134.

Yin, R. K. (1969). Looking at upside-down faces. Journal of ExperimentalPsychology, 81, 141–145.

Young, A. W., Hellawell, D., & Hay, D. C. (1987). Configurationalinformation in face perception. Perception, 16, 747–759.

Yovel, G., Pelc, T., & Lubetzky, I. (2010). It’s all in your head: Why is thebody inversion effect abolished for headless bodies? Journal ofExperimental Psychology: Human Perception and Performance, 36,759–767.

Neuron, Vol. 41, 465–472, February 5, 2004, Copyright 2004 by Cell Press

The Neural Fate of Consciously Perceivedand Missed Events in the Attentional Blink

second target (T2) is a result of attending to the firsttarget (T1): subjects have no difficulties in reporting T2when it is the only target to be detected (Joseph et al.,

Rene Marois,1,* Do-Joon Yi,1,2

and Marvin M. Chun1,2

1Vanderbilt Vision Research Center1997; Raymond et al., 1992). Thus, T2 can easily beCenter for Integrative and Cognitive Neurosciencesingled out of an RSVP of distractor items, unless atten-Department of Psychologytion is engaged in processing a previously presentedVanderbilt Universitytarget (T1). These results support a two-stage model of530 Wilson Hallthe AB, consisting of the rapid and initial representationNashville, Tennessee 37203of visual items followed by the slow, capacity-limitedand attention-demanding consolidation of these itemsfor conscious report (Chun and Potter, 1995; JolicoeurSummaryet al., 2001; Shapiro et al., 1997b).

Although such a dual mode of visual information pro-Cognitive models of attention propose that visual per-cessing figures prominently in cognitive models of theception is a product of two stages of visual processing:AB and of attention in general, it is not yet clear whetherearly operations permit rapid initial categorization ofit also characterizes the underlying functional neuroar-the visual world, while later attention-demanding ca-chitecture. In support of a first stage of information pro-pacity-limited stages are necessary for the consciouscessing, there is both electrophysiological (Luck et al.,report of the stimuli. Here we used the attentional blink1996) and behavioral (Shapiro et al., 1997a) evidenceparadigm and fMRI to neurally distinguish these twothat visually presented words which are not explicitlystages of vision. Subjects detected a face target andperceived in the attentional blink are nonetheless pro-a scene target presented rapidly among distractors atcessed up to their semantic identity. However, thesefixation. Although the second, scene target frequentlystudies could not determine the functional neuroanat-went undetected by the subjects, it nonetheless acti-omy of unconsciously processed events under condi-vated regions of the medial temporal cortex involvedtions of inattention nor could they reveal how it differsin high-level scene representations, the parahippo-from that of consciously perceived events. In supportcampal place area (PPA). This PPA activation was am-of a second, attention-demanding stage, manipulationsplified when the stimulus was consciously perceived.that affect the magnitude of the AB recruit a parietofron-By contrast, the frontal cortex was activated only whental cortical network (Marois et al., 2000a) previously im-scenes were successfully reported. These results sug-plicated in the control of visuospatial attention (Corbettagest that medial temporal cortex permits rapid catego-et al., 1993, 1998; Kastner et al., 1999; Nobre et al.,rization of the visual input, while the frontal cortex1997). However, the Marois et al. (2000a) study focusedis part of a capacity-limited attentional bottleneck toon neural processing of T1, namely, the attentional limi-

conscious report.tations that cause the AB. The present study now exam-ines the effects of divided attention on T2, both when

Introduction it is consciously perceived and when it is missed.Few imaging studies have investigated the neural fate

Virtually all cognitive models of attention posit that hu- of consciously perceived and missed visual events un-man perception proceeds along at least two stages der conditions of divided attention, and they have(Broadbent and Broadbent, 1987; Chun and Potter, yielded inconsistent results. One study reported no evi-1995; Duncan, 1980; Neisser, 1967; Rensink, 2002; Shif- dence that foveally presented words are semanticallyfrin and Gardner, 1972; Treisman and Gelade, 1980; processed by the brain in the absence of attention (ReesWolfe et al., 1989). The first stage of perceptual analysis et al., 1999). Although another observed distinct inferioris considered to be largely unconscious and allows for temporal and parietofrontal activation patterns for con-the rapid, global, and highly efficient categorization of sciously and unconsciously perceived face changesitems and events in a visual scene. The second “atten- (Beck et al., 2001), that study could not distinguish be-tional” stage is necessary for the thorough identification, tween neural activity associated with awareness of theconsolidation, and conscious report of visual events. change versus activity associated with spatial shifts of

The dual nature of perception is clearly illustrated attention toward the change since the objects were notby the attentional blink (AB) paradigm: when subjects presented at the focus of attention.search for two targets presented in a rapid serial visual Thus, the goal of the present study is to determinedisplay of distractor items, they are severely impaired whether the two stages of visual information processing

predicted by cognitive models of attention are imple-at detecting the second of the two targets when it ismented by different neural substrates under experimen-presented within 500 ms of the first target (Chun andtal conditions that eliminate contributions of spatialPotter, 1995; Raymond et al., 1992). The deficit with theshifts of attention. Specifically, we used an AB paradigmto test whether the neural activation associated with*Correspondence: rene.marois@vanderbilt.educonsciously reported and unreported targets presented2 Present address: Department of Psychology, Yale University, 2at fixation is different in regions of the inferior temporalHillhouse Avenue, P.O. Box 208205, New Haven, Connecticut

06520-8205. cortex involved in visual categorization and representa-

Neuron466

Figure 1. Experimental Design

In the dual-task experiment, subjectssearched for a face target (T1) and a scenetarget (T2) presented in an RSVP of scram-bled distractor scenes. The SOA between T1and T2 was varied. The single-task experi-ment was identical except that subjectssearched only for the target scene. Insetsshow the three face targets and examples ofboth indoor and outdoor scene targets.

tion, than in the parietofrontal cortical network pre- below) but not enough to obscure intact scene-relatedactivity, as the PPA activates significantly more to intactviously hypothesized to represent a capacity-limited at-

tention-demanding stage (Marois et al., 2000a). The than scrambled scenes (Epstein and Kanwisher, 1998).Although in principle T1-related activity could alsohypothesis that the AB bottleneck occurs at a late locus

of processing predicts that (1) both reported and unre- be examined, since face stimuli activate a well-definedregion of the fusiform gyrus (Kanwisher et al., 1997;ported visual items should engage even high-level

stages of visual event representation in ventral regions Sergent et al., 1992), this is not feasible in the presentexperiment since the face-sensitive area also respondsof the occipitotemporal cortex (Treisman and Kan-

wisher, 1998; Malach et al., 2002) and that (2) the neural to buildings/scenes (Ishai et al., 2000; Kanwisher et al.,1997), thereby preventing the independent assessmentdistinction between reportable and unreportable items

should occur later along the information processing of T1-related brain activity from T2 performance andactivity. Instead, the design of the experiment and thepathway, specifically in the parietofrontal network of

visuospatial attention (Beck et al., 2001; Marois et al., results described below focus on isolating the neuralresponse to T2 processing.2000a).

Results Behavioral ExperimentA behavioral experiment performed outside the scannerroom established that an attentional blink can be ob-The task consisted of searching for two targets pre-

sented among a 1 s long rapid serial display of scram- tained with this experimental paradigm (Figure 2). Scenedetection performance was substantially lower whenbled scenes (Figure 1). The first target (T1) was a face,

the second (T2) a scene, and distractors consisted of subjects were required to detect both T1 and T2 thanscrambled versions of scenes. This design bestowedseveral crucial experimental advantages for this study.First, since the scenes activate a canonical region ofthe visual cortex, namely the parahippocampal placearea (PPA) (Epstein and Kanwisher, 1998), the brain re-sponse to T2 stimulus presentations can be easily local-ized. Second, the use of scenes as T2 and faces as T1permits the assessment of the brain response to scenesuncontaminated by the processing of T1, since facesproduce negligible activation of the PPA (Epstein et al.,1999; Epstein and Kanwisher, 1998), a finding we con-firmed in pilot scanning sessions (data not shown). Third,since the PPA is involved in high-level perception,namely in the perceptual encoding (Epstein et al., 1999)and representation (Epstein et al., 2003) of visual scenes,it is ideally suited to test whether high-level visual repre- Figure 2. Behavioral T2 Performance under Single- and Dual-sentation can occur even in the absence of conscious Task Conditionsreport. Finally, scrambled scenes conceal intact scenes T2 performance was worse in the dual-task than in the single-task

condition, especially at small SOAs. Error bars: � SEM.sufficiently to render detection of the latter difficult (see

The Neural Fate of Events in the Attentional Blink467

when they were required to detect only T2 [F(1,16) �22.7, p � 0.001, ANOVA with condition (single/dual task)as between-subject and SOA as within-subject factors].Furthermore, in the dual-task condition, performanceincreased with greater SOA between T1 and T2 [F(2,32) �15.4, p � 0.001]. These two results are trademark fea-tures of the AB (Chun and Potter, 1995; Raymond etal., 1992).

fMRI Experiment: Behavioral PerformanceA similar dual-task experiment was carried out in thescanner. Mean T1 accuracy was 86%, with a 2% falsealarm rate. T2 accuracy was experimentally held around50% detection to yield similar number of trials in T2-detected and T2-undetected conditions by adjusting the

Figure 3. Timecourse of the Hemodynamic Response in the Para-T1-T2 SOA between fMRI runs (mean T1-T2 SOA: 450 hippocampal Place Area under Hit, Miss, and CR T2 Conditionsms). Presumably, for any set SOA, whether T2 is de- Error bars: � SEM.tected or missed on any given trial is governed by sto-chastic variations in the activity levels of the neural sub-strates involved in T1 and T2 identifications (Dehaene by the subjects (Miss) still activated the PPA more thanet al., 2003). With this SOA manipulation, T2 was de- when no scenes were presented (CR) [Miss � CR, t(18) �tected on 52% (27% correctly and 25% incorrectly iden- 2.19, p � 0.05], suggesting that the PPA responds totified scenes) of the trials and was missed on 48%. The scenes even when they are not consciously perceived.mean T2 false alarm rate was 24%, which was signifi- Moreover, this subliminal PPA activation was enhancedcantly below the T2 detection rate [F(1,18) � 8.920, p � when subjects consciously perceived the scenes [Figure001]. Finally, as expected, subjects performed very well 3; Hit � Miss, t(18) � 2.31, p � 0.05], suggesting thatwith T2 (82% accuracy) in trials where T1 was absent, conscious scene perception amplified the PPA re-demonstrating again that T2 performance is impaired sponse elicited by subliminal scene perception.by attention to T1. Parietofrontal Cortex. The results in the PPA suggest

that the medial temporal cortex discriminates betweenscenes and nonscenes even when these are not con-fMRI Experiment: T2-Related Brain Activations

Medial Temporal Cortex. The PPA region of each subject sciously perceived by the subjects under conditions ofdivided attention. Based on previous findings (Maroiswas first isolated in a localizer task by contrasting the

brain activity in blocked presentations of faces and et al., 2000a), we postulated that a network of lateralfrontal, anterior cingulate (AC), and intraparietal areasscenes. The mean Talairach coordinates of the isolated

region (right PPA: x � 21.4 mm, y � �53.1 mm, z � may represent the attentional bottleneck to perceptualawareness. This hypothesis predicts that the parieto-�5.38 mm; left PPA: x � �22.8 mm, y � �56.6 mm, z �

�5.38 mm) is consistent with the known location of the frontal network should respond differently than themedial temporal cortex under the three T2 conditions.PPA (Epstein et al., 2003; Epstein and Kanwisher, 1998).

The isolated PPAs from both left and right hemi- Specifically, activity in this network should be the com-parable in Miss and CR trials, whereas conscious scenespheres were collapsed and probed in the dual-task

experiment for scene-related activity under three differ- detection (Hit) should recruit these brain regions morethan either of the two other conditions. A voxel-basedent T2 performance conditions: (1) subjects successfully

detected the presentation of a scene (Hit), regardless approach did not reveal any parietofrontal regions thatwere significantly more activated in the Hit than in theof whether the scene was correctly identified as indoor

or outdoor (see Experimental Procedures); (2) subjects Miss and CR conditions (p � 0.05, corrected). However,a more sensitive ROI approach, using regions of interestfailed to detect the scene (Miss), and (3) subjects cor-

rectly reported the absence of a scene (CR). Too few defined in a previous AB study (Marois et al., 2000a),revealed activation in part of this parietofrontal networkfalse alarm trials (scene was reported when none was

shown) were obtained per subject (mean � 7.4) to yield with T2 response (Figures 4A and 4B). Specifically, thebilateral frontal cortex activation conformed [F(1,18) �a stable response for this condition. The PPA was acti-

vated even when no scenes were presented and de- 5.12, p � 0.05] to the predicted response function(ANOVA with 2Hit, -1Miss, -1CR contrast weights), andtected (CR, Figure 3). This activation likely resulted from

the entire sequence of scrambled scenes since we ob- the anterior cingulate (AC) showed a similar though mar-ginal effect [F(1,18) � 4.39, p � 0.051]. These resultsserved in preliminary scanning sessions similar re-

sponses even when neither any scenes nor faces were were also generally supported by paired t tests analysisfor both the lateral frontal (Miss-CR p � 0.339, Hit-CRpresented (data not shown), confirming that scrambled

scenes activate the PPA (Epstein and Kanwisher, 1998). p � 0.05, Hit-Miss p � 0.054) and AC (Miss-CR p �0.068, Hit-CR p � 0.05, Hit-Miss p � 0.221). The greaterThe CR condition provides the baseline on which the

other two conditions can be compared. As expected, response in the frontal cortex with Hits than with Missesand CRs is evidenced not only by higher peak amplitudethe PPA was more activated when subjects detected

the presence of a scene [Hit � CR, t(18) � 4.38, p � but also by a prolonged response (Figure 4A).In contrast to the frontal areas, the parietal ROI did.001]. Most importantly, scenes that were not detected

Neuron468

Figure 4. Timecourse of the Hemodynamic Response in the Parietal and Frontal Cortex under Hit, Miss, and CR T2 Conditions

(A) Intraparietal cortex, Talairach coordinates (x, y, z) of the ROI centroid (Marois et al., 2000a): �30, �58, �45; (B) lateral frontal cortex, �48,�8, �35; (C) anterior cingulate, �3, �20, �36; (D) right temporoparietal junction, �53, �34, �21. Error bars: � SEM.

not show significant activation differences between any jects fail to consciously perceive foveated stimuli underdivided attention. By contrast, the frontal cortex’s re-of the three conditions [F(1,18) � 0.12, p � 0.913 for the

linear contrast analysis, ps � 0.25 for pair-wise t tests], sponse to the stimulus is primarily contingent on whetherthat stimulus is consciously reported by the subject.although it did show a prolonged response as well (Fig-

ure 4C). Finally, given recent reports that patients with Thus, activity in the inferior/medial temporal cortex pri-marily reflects the physical visual world, while the frontallesions in the temporoparietal cortex may exhibit abnor-

mally long ABs (Husain et al., 1997; Shapiro et al., 2002), cortex predominantly represents the consciously re-ported world.we also examined this region (Marois et al., 2000b) and

found no systematic differences between the three con-ditions (Figure 4D), either with the contrast analysis [right Parahippocampal Cortex

The greater activation of the PPA in Miss than in CRTPJ, F(1,18) � 0.407, p � 0.53; left TPJ, F(1,18) � 0.456,p � 0.508] or pair-wise t tests (all ps � 0.05). trials suggests that the visual cortex can categorize vi-

sual input under conditions of high attentional load thatOverall, these results indicate that, unlike the medialtemporal cortex, the frontal cortex activation is mainly prevents awareness and report. These results are con-

sistent with behavioral and electrophysiological workdictated by the subjects’ explicit perception of the stim-ulus rather than by its physical presentation. Impor- suggesting that stimuli that fail to be explicitly reported

during the AB are nevertheless registered by the braintantly, the distinct activation pattern in the frontal andmedial temporal cortex argues against a simple account (Luck et al., 1996; Shapiro et al., 1997a) but inconsistent

with the idea that the brain is unresponsive to stimuliof detection bias for the results in the PPA, i.e., thatactivation in Miss trials might not be due to processing that the mind is inattentive to (Rees et al., 1999). This

finding supports the view that visual cortex activationof unattended scenes but instead to subjects adoptinga conservative criterion for the report of the target scene, is not sufficient for visual awareness (Beck et al., 2001;

Dehaene et al., 2001) even when the stimuli are foveated.leading them to classify trials for which they were uncer-tain about the presence of a scene as Miss. Since this They also demonstrate that, unlike previous reports

(Beck et al., 2001), scenes can activate the medial tem-bias is not reflected in the activity of the frontal cortex,where decision making is thought to be more prevalent poral cortex during inattention, raising the possibility

that scenes are automatically categorized by the PPA.than in visual cortex (Gold and Shadlen, 2001; Schall,2001), it is unlikely to account for the PPA activity. Importantly, these results do not imply that the medial

temporal cortex is not critical for conscious, attentiveperception of the visual world, as has been evidencedDiscussionwith brain lesion (Farah and Feinberg, 1997) and physio-logical studies (Bar et al., 2001; Kleinschmidt et al., 2002;The findings of this study clearly establish that the me-

dial temporal cortex can be activated even when sub- Logothetis, 1998; Lumer et al., 1998; Moutoussis and

The Neural Fate of Events in the Attentional Blink469

Zeki, 2002; Pins and Ffytche, 2003; Tong et al., 1998). the intraparietal cortex appears to be primarily engagedby temporal and spatial changes of attentional demandsFurthermore, the visual cortex in general (Chawla et al.,

1999; Kastner et al., 1998; Luck et al., 1997; Spitzer (Corbetta et al., 1993; Coull and Nobre, 1998; Yantis etal., 2002). Since the attentional demands were constantet al., 1988), and the PPA in particular (O’Craven and

Kanwisher, 2000; see also Figure 3), is strongly modu- across all T2 conditions in the current experiment, thishypothesis would predict little activation differencelated by attention and imagery. Correspondingly, PPA

activity was enhanced above and beyond the activation among these conditions. The attentional demand hy-pothesis is also consistent with the observation of IPSlevels of the Miss condition when subjects consciously

perceived the scenes (Figure 3), perhaps as a result of activation with detection of scene changes (Beck et al.,2001), since the detection of a change may lead to a shiftattentional top-down modulation of the PPA with scene

detection. Taken together, these results clearly indicate of visuospatial attention to the location of the change.that PPA activity represents a conflation of automatic/bottom-up and conscious/top-down sources of acti- Neural Substrates of the Attentional Blinkvation. The attentional blink reveals a central processing limita-

tion in attending to two targets presented in an RSVPof distractor items. Consistent with a central limitation,Parietofrontal Cortex

In contrast to the medial temporal cortex, the lateral the AB is a robust phenomenon that has been observedwith a wide variety of target objects and events (Josephfrontal cortex activation was strongly dependent on

whether the target scenes were explicitly reported. et al., 1997; Ross and Jolicoeur, 1999; Shapiro et al.,1997b; Sheppard et al., 2002). Our behavioral resultsThese results are consistent with the involvement of

this brain region in the control of visuospatial attention extend the generality of the attentional blink in demon-strating that it not only applies to the perception of(Corbetta et al., 1993; Kastner et al., 1999; Nobre et al.,

1997) and suggest that the frontal cortex is particularly objects but to the perception of complex scenes as well.As such, these results challenge a recent finding thatimportant for conscious target report (Beck et al., 2001;

Dehaene et al., 2001; Weiskrantz et al., 2003). The pre- scenes can be overtly categorized in the absence ofattention (Li et al., 2002). In contrast to their null finding,cise function played by the lateral frontal cortex in the

present task remains to be determined, although it is we observed pronounced scene detection deficits,probably because our procedures—namely, the sus-likely to be associated with some aspects of reporting

the conscious perception of the target, such as the con- tained RSVP task and robust masking of the targetscenes by scrambled scene distractors—were more ef-solidation and maintenance of the target in working

memory for subsequent report (Courtney et al., 1998). fective at taxing attention.The results also provide neural support for two-stageConsistent with an involvement in working memory, the

frontal cortex showed a prolonged hemodynamic re- models of visual attention. More specifically, the two-stage model of the attentional blink proposed that stim-sponse with hits relative to misses or correct rejections

(Figure 4), which may reflect further decision making uli are initially characterized and registered at an earlystage of visual information processing, but explicit re-about scene category (indoor/outdoor) following an ini-

tial judgment about the presence or absence of a scene. port of the stimuli requires attentional consolidation ofthe stimuli into a durable form (working memory) (ChunThis is a testable hypothesis, since one would predict

that larger but not prolonged responses should be ob- and Potter, 1995; Jolicoeur and Dell’Acqua, 1998, Vogelet al., 1998). This model echoes other attention modelsserved when subjects are only asked to perform a judg-

ment about the presence or absence of a scene. The that distinguish between efficient, preconscious andmore capacity-limited, attention-demanding stages ofanterior cingulate cortex showed a similar response

trend to the lateral frontal cortex. Viewed in the light of vision (Broadbent and Broadbent, 1987; Duncan, 1980;Neisser, 1967; Rensink, 2002; Shiffrin and Gardner,the involvement of the AC in response conflict and/or

performance/error monitoring (Carter et al., 1998; Gehr- 1972; Treisman and Gelade, 1980; Wolfe et al., 1989).Consistent with this two-stage progression of atten-ing et al., 1993; Paus, 2001), it is possible that the AC

activation in this study may be response related, per- tional processing, the present results demonstrate dif-ferent response patterns in visual and frontal cortex:haps reflecting indecision or monitoring processes.

The response of the intraparietal cortex did not distin- the lateral frontal cortex is activated when subjects cansuccessfully report the target, while high-level visualguish between the different T2 conditions, although it

showed the same trend of prolonged Hit activation ob- cortex still registers the visual stimuli even when theyare not reportable. It should be noted that these findingsserved in the frontal cortex. This suggests that the IPS

may not be as involved in conscious target report as do not imply that the two stages of information pro-cessing with the attentional blink are necessarily dis-the lateral frontal cortex. On the other hand, T1 manipu-

lations of perceptual interference that affect the magni- crete, as the results are not inconsistent with gradedmodels of activation to awareness. Our experimentaltude of the AB modulate IPS activation (Marois et al.,

2000a). It is therefore conceivable that the parietal cortex design and analysis may simply reveal extremes of acontinuum, although recent modeling and behavioralis important for resolving perceptual interference (Fried-

man-Hill et al., 2003; Marois et al., 2000a; Shafritz et al., evidence suggests that the attentional blink may resultfrom a nonlinear transition from nonconscious pro-2002; Wojciulik and Kanwisher, 1999) or, more broadly

speaking, in controlling the distribution of attentional cessing to explicit perception (Dehaene et al., 2003).More broadly speaking, the frontal cortex may be as-resources among visual events, rather than in explicit

target perception per se. In support of this hypothesis, sociated with capacity-limited attentive vision, while the

Neuron470

BOLD activation for each subject were created using a skew-cor-visual cortex registers the input in an efficient, precon-rected percent signal difference. The PPA ROI was defined as thescious manner that guides selection for report (Chunvoxel with the peak activation and its eight surrounding voxels, suchand Marois, 2002; Marois et al., 2000a). Accordingly,that each subject provided a 3 � 3 voxel grid from each hemisphere.

activation of the visual cortex is not sufficient for con- For all subjects, the activated region was found in the parahippo-scious vision, which would necessitate the recruitment campal gyrus/collateral sulcus region.

Dual Taskof the frontal cortex (Beck et al., 2001; Dehaene et al.,Subjects subsequently performed four to eight runs of an event-2001, 2003; Lumer et al., 1998; Rees et al., 2002). Clearly,related dual-task experiment similar to the behavioral experimentthe explicit perception of a visual stimulus is likely toexcept for the following modifications. The response panels were

result from the interaction between a sensory represen- followed by a 12.7 s fixation period and by a 1000 ms blank period,tation of the visual stimulus in visual cortex and the which signaled the beginning of the next trial (total trial duration �attentional network necessary to consolidate that stimu- 18 s). Nineteen trials were presented in each fMRI run, including

five T1-only trials and four T2-only trials. After each run, the temporallus for full report in the frontal cortex.lag between the T1 face target and the T2 scene target was adjustedby the experimenter in order to yield a scene detection performance

Experimental Proceduresaround 50%, and the hit and false alarm rates for the face task weregiven to subjects as feedback. Unlike for the behavioral experiment,

Behavioral Experiments“no_face” and “unknown_scene” response options were added for

Nine paid subjects from the Vanderbilt University community volun- the T1 and the T2 task, respectively. The “unknown_scene” re-teered for each of the single- and dual-task experiments. In the sponse was included in case subjects perceived the layout of adual-task experiment, subjects searched for two targets presented scene but were not certain whether it was indoor or outdoor. Foramong an RSVP of eight distractor items at fixation for 100 ms each data analysis, selection of this response option was classified aswith no interstimulus interval. The first target (T1) was a face, the an incorrect scene identification, where it accounted for 64% of thissecond (T2) a scene, and the distractors were scrambled versions category’s trials. However, since all ROIs showed indistinguishableof scenes, with each grayscale stimulus subtending 12.8� � 12.8�. responses to correctly identified and incorrectly identified scenesThe scrambled scenes originated from a pool of 120 scenes and (data not shown), these two responses were combined into thewere created by dividing each quadrant of the image into 25 squares, category of correct scene detection.inverting these squares, and randomly scrambling their positions. Data AnalysisThin black grids were drawn over the scrambled (and intact) images One predictable consequence of the lag manipulation for keepingto occlude the boundaries of blocks. When present, the scene target subjects’ T2 performance around 50% is that it led to a differencewas shown at the second-to-last position in the RSVP, while the (t test, p � 0.05) in T1-T2 SOA between the Hit (mean SOA: 452 ms)face target was presented 200, 400, or 800 ms before the scene and Miss (mean SOA: 435 ms) conditions. Even though this differ-target. A trial began with presentation of a fixation point for 1200 ence in mean stimulus onset asynchronies is small, to prevent suchms before the onset of the RSVP and ended with the presentation differences in stimulus presentations from influencing the activationof both T1 response and T2 response displays, each for 1800 ms. differences between Hits and Misses, we equated the Hits andDuring the T1 displays (labeled “Face1_Face2_Face3”), subjects Misses SOAs by extracting fMRI runs which showed the longest lagdecided by keypress which of the three faces was presented, while as well as the greatest hit rate, or the shortest lag as well as theduring the T2 response displays (“NoScene_Indoor_Outdoor”) they greatest miss rate. Nine runs from eight subjects were thereby elimi-selected whether no scene, an outdoor scene, or an indoor scene nated from further analysis. One subject was removed from further

analysis, as the resulting number of CR trials was excessively low (awas presented. A face target was present on every trial and a scenepriori criterion that subjects with fewer than eight trials per conditiontarget on 67% of the trials, with equal probability of indoor andwould be discarded). The group average SOA for Hits and Missesoutdoor scene presentation. When absent, T2 targets were replacedwere no longer significantly different from each other (SOA differ-by a scrambled scene. Subjects were instructed to emphasize taskence: 9 ms, p � 0.155). The SOA difference between Correct Rejec-1 over task 2. For T2 performance, only T1-correct trials were ana-tion and Miss trials was also not significant (SOA difference: 4 ms,lyzed. The single-task experiment was identical to the dual-taskp � 0.220).experiment except that subjects were instructed to search for the

For each ROI of each subject, the BOLD percent change wasscene target only. A total of 180 trials were presented in each exper-calculated by averaging the time courses of each T2 condition (Hit,iment.Miss, CR) and normalizing them to the averaged value of the firsttwo TRs after trial onset (Figure 3). ROI time courses were then

fMRI Experiment averaged across all subjects. Statistical analysis (paired t tests andTwenty paid subjects (9 females) from the Vanderbilt University contrast analysis) between conditions was performed on the peakcommunity performed a similar dual-task in an fMRI experiment. amplitude response (Epstein et al., 2003), the time point of whichThe 12.8� � 12.8� stimuli were viewed by the subjects on a projection was first determined by collapsing all T2 conditions together. Anscreen through a mirror mounted on top of the RF coil above their area under the curve (AUC) analysis yielded qualitatively similarhead. Stimuli were projected onto the screen by means of an LCD results to the peak analysis. Only T1-correct trials were examinedprojector located outside the scanner room. for T2-related brain activity.fMRI ParametersSubjects were scanned on a 3T whole-body GE/Magnex (Milwau- Acknowledgmentskee, WI). Anatomical images were acquired using conventional pa-rameters. T2* scan parameters: TR 2 s, TE 25 ms, FA 70�, 197 We wish to thank Todd Kelley for technical assistance and Ye-Seulimages/slice, with 19 axial slices (7 mm thick, 0 mm skip) acquired Choi for assistance with data analysis. This work was supported byparallel to the AC-PC line. NSF grant #0094992 and in part by NIH R01 EY014193.Localizer TaskSubjects were first presented with two runs of a one-back repetition Received: September 10, 2003detection task in order to localize the PPA (Epstein et al., 1999, Revised: November 25, 20032003; Levy et al., 2001). The design consisted of alternating blocked Accepted: December 29, 2003presentation of faces and scenes, with each block containing 18 Published: February 4, 2004scenes or faces presented at fixation for 800 ms followed by a blankof 200 ms. Subjects searched for consecutive repetitions of stimuli, Referenceswith two such repetitions occurring in each block. There were nineblocks each of faces and scenes in one fMRI run. The PPA was Bar, M., Tootell, R.B., Schacter, D.L., Greve, D.N., Fischl, B., Men-localized in each individual by contrasting the averaged brain activity dola, J.D., Rosen, B.R., and Dale, A.M. (2001). Cortical mechanisms

specific to explicit visual object recognition. Neuron 29, 529–535.in scene blocks with face blocks. Statistical parametric maps of

The Neural Fate of Events in the Attentional Blink471

Beck, D.M., Rees, G., Frith, C.D., and Lavie, N. (2001). Neural corre- tional blink bottleneck. In The Limits of Attention: Temporal Con-straints in Human Information Processing. K. Shapiro, ed. (New York:lates of change detection and change blindness. Nat. Neurosci.

4, 645–650. OU Press), pp. 82–99.

Joseph, J.S., Chun, M.M., and Nakayama, K. (1997). AttentionalBroadbent, D.E., and Broadbent, M.H. (1987). From detection torequirements in a “preattentive” feature search task. Nature 387,identification: Response to multiple targets in rapid serial visual805–808.presentation. Percept. Psychophys. 42, 105–113.

Kanwisher, N., McDermott, J., and Chun, M.M. (1997). The fusiformCarter, C.S., Braver, T.S., Barch, D.M., Botvinick, M.M., Noll, D., andface area: a module in human extrastriate cortex specialized forCohen, J.D. (1998). Anterior cingulate cortex, error detection, andface perception. J. Neurosci. 17, 4302–4311.the online monitoring of performance. Science 280, 747–749.

Kastner, S., De Weerd, P., Desimone, R., and Ungerleider, L.G.Chawla, D., Rees, G., and Friston, K.J. (1999). The physiological(1998). Mechanisms of directed attention in the human extrastriatebasis of attentional modulation in extrastriate visual areas. Nat. Neu-cortex as revealed by functional MRI. Science 282, 108–111.rosci. 2, 671–676.

Kastner, S., Pinsk, M.A., De Weerd, P., Desimone, R., and Unger-Chun, M.M., and Marois, R. (2002). The dark side of visual attention.leider, L.G. (1999). Increased activity in human visual cortex duringCurr. Opin. Neurobiol. 12, 184–189.directed attention in the absence of visual stimulation. Neuron 22,Chun, M.M., and Potter, M.C. (1995). A two-stage model for multiple751–761.target detection in rapid serial visual presentation. J. Exp. Psychol.Kleinschmidt, A., Buchel, C., Hutton, C., Friston, K.J., and Frackow-Hum. Percept. Perform. 21, 109–127.lak, R. (2002). The neural structures expressing perceptual hystere-Corbetta, M., Miezin, F.M., Shulman, G.L., and Petersen, S.E. (1993).sis in visual letter recognition. Neuron 34, 659–666.A PET study of visuospatial attention. J. Neurosci. 13, 1202–1226.Levy, I., Hasson, U., Avidan, G., Hendler, T., and Malach, R. (2001).Corbetta, M., Akbudak, E., Conturo, T.E., Snyder, A.Z., Ollinger, J.M.,Center-periphery organization of human object areas. Nat. Neurosci.Drury, H.A., Linenweber, M.R., Petersen, S.E., Raichle, M.E., Van4, 533–539.Essen, D.C., and Shulman, G.L. (1998). A common network of func-Li, F.F., VanRullen, R., Koch, C., and Perona, P. (2002). Rapid naturaltional areas for attention and eye movements. Neuron 21, 761–773.scene categorization in the near absence of attention. Proc. Natl.Coull, J.T., and Nobre, A.C. (1998). Where and when to pay attention:Acad. Sci. USA 99, 9596–9601.the neural systems for directing attention to spatial locations andLogothetis, N. (1998). Object vision and visual awareness. Curr.to time intervals as revealed by both PET and fMRI. J. Neurosci.Opin. Neurobiol. 8, 536–544.18, 7426–7435.

Luck, S.J., Vogel, E.K., and Shapiro, K.L. (1996). Word meaningsCourtney, S.M., Petit, L., Haxby, J.V., and Ungerleider, L.G. (1998).can be accessed but not reported during the attentional blink. NatureThe role of prefrontal cortex in working memory: examining the383, 616–618.contents of consciousness. Philos. Trans. R Soc. Lond. B Biol. Sci.

353, 1819–1828. Luck, S.J., Chelazzi, L., Hillyard, S.A., and Desimone, R. (1997).Neural mechanisms of spatial selective attention in areas V1, V2,Dehaene, S., Naccache, L., Cohen, L., Bihan, D.L., Mangin, J.F.,and V4 of macaque visual cortex. J. Neurophysiol. 77, 24–42.Poline, J.B., and Riviere, D. (2001). Cerebral mechanisms of word

masking and unconscious repetition priming. Nat. Neurosci. 4, Lumer, E.D., Friston, K.J., and Rees, G. (1998). Neural correlates of752–758. perceptual rivalry in the human brain. Science 280, 1930–1934.Dehaene, S., Sergent, C., and Changeux, J.P. (2003). A neuronal Malach, R., Levy, I., and Hasson, U. (2002). The topography of high-network model linking subjective reports and objective physiological order human object areas. Trends Cogn. Sci. 6, 176–184.data during conscious perception. Proc. Natl. Acad. Sci. USA Marois, R., Chun, M.M., and Gore, J.C. (2000a). Neural correlates100, 8520–8525. of the attentional blink. Neuron 28, 299–308.Duncan, J. (1980). The locus of interference in the perception of Marois, R., Leung, H.C., and Gore, J.C. (2000b). A stimulus-drivensimultaneous stimuli. Psychol. Rev. 87, 272–300. approach to object identity and location processing in the humanEpstein, R., and Kanwisher, N. (1998). A cortical representation of brain. Neuron 25, 717–728.the local visual environment. Nature 392, 598–601. Moutoussis, K., and Zeki, S. (2002). The relationship between corti-Epstein, R., Harris, A., Stanley, D., and Kanwisher, N. (1999). The cal activation and perception investigated with invisible stimuli.parahippocampal place area: recognition, navigation, or encoding? Proc. Natl. Acad. Sci. USA 99, 9527–9532.Neuron 23, 115–125. Neisser, U. (1967). Cognitive Psychology (New York: Appleton-Cen-Epstein, R., Graham, K.S., and Downing, P.E. (2003). Viewpoint- tury-Crofts).specific scene representations in human parahippocampal cortex. Nobre, A.C., Sebestyen, G.N., Gitelman, D.R., Mesulam, M.M.,Neuron 37, 865–876. Frackowiak, R.S., and Frith, C.D. (1997). Functional localization ofFarah, M.J., and Feinberg, T.E. (1997). Consciousness of perception the system for visuospatial attention using positron emission tomog-after brain damage. Semin. Neurol. 17, 145–152. raphy. Brain 120, 515–533.

Friedman-Hill, S.R., Robertson, L.C., Desimone, R., and Ungerleider, O’Craven, K.M., and Kanwisher, N. (2000). Mental imagery of facesL.G. (2003). Posterior parietal cortex and the filtering of distractors. and places activates corresponding stimulus-specific brain regions.Proc. Natl. Acad. Sci. USA 100, 4263–4268. J. Cogn. Neurosci. 12, 1013–1023.

Gehring, W.J., Goss, B., Coles, M.G.H., Meyer, D.E., and Donchin, Paus, T. (2001). Primate anterior cingulate cortex: where motor con-E. (1993). A neural system for error detection and compensation. trol, drive and cognition interface. Nat. Rev. Neurosci. 2, 417–424.Psychol. Sci. 4, 385–390. Pins, D., and Ffytche, D. (2003). The neural correlates of consciousGold, J.I., and Shadlen, M.N. (2001). Neural computations that un- vision. Cereb. Cortex 13, 461–474.derlie decisions about sensory stimuli. Trends Cogn. Sci. 5, 10–16. Raymond, J.E., Shapiro, K.L., and Arnell, K.M. (1992). TemporaryHusain, M., Shapiro, K., Martin, J., and Kennard, C. (1997). Abnormal suppression of visual processing in an RSVP task: An attentionaltemporal dynamics of visual attention in spatial neglect patients. blink? J. Exp. Psychol. Hum. Percept. Perform. 18, 849–860.Nature 385, 154–156. Rees, G., Russell, C., Frith, C.D., and Driver, J. (1999). InattentionalIshai, A., Ungerleider, L.G., Martin, A., and Haxby, J.V. (2000). The blindness versus inattentional amnesia for fixated but ignoredrepresentation of objects in the human occipital and temporal cor- words. Science 286, 2504–2507.tex. J. Cogn. Neurosci. 12 (Suppl 2), 35–51. Rees, G., Kreiman, G., and Koch, C. (2002). Neural correlates ofJolicoeur, P., and Dell’Acqua, R. (1998). The demonstration of short- consciousness in humans. Nat. Rev. Neurosci. 3, 261–270.term consolidation. Cognit. Psychol. 36, 138–202. Rensink, R.A. (2002). Change detection. Annu. Rev. Psychol. 53,

245–277.Jolicoeur, P., Dell’Acqua, R., and Crebolder, J.M. (2001). The atten-

Neuron472

Ross, N.E., and Jolicoeur, P. (1999). Attentional blink for color. J.Exp. Psychol. Hum. Percept. Perform. 25, 1483–1494.

Schall, J.D. (2001). Neural basis of deciding, choosing and acting.Nat. Rev. Neurosci. 2, 33–42.

Sergent, J., Ohta, S., and MacDonald, B. (1992). Functional neuro-anatomy of face and object processing. A positron emission tomog-raphy study. Brain 115, 15–36.

Shafritz, K.M., Gore, J.C., and Marois, R. (2002). The role of theparietal cortex in visual feature binding. Proc. Natl. Acad. Sci. USA99, 10917–10922.

Shapiro, K., Driver, J., Ward, R., and Sorensen, R.E. (1997a). Primingfrom the attentional blink: A failure to extract visual tokens but notvisual types. Psychol. Sci. 8, 95–100.

Shapiro, K.L., Arnell, K.M., and Raymond, J.E. (1997b). The atten-tional blink. Trends Cogn. Sci. 1, 291–296.

Shapiro, K., Hillstrom, A.P., and Husain, M. (2002). Control of visuo-temporal attention by inferior parietal and superior temporal cortex.Curr. Biol. 12, 1320–1325.

Sheppard, D.M., Duncan, J., Shapiro, K.L., and Hillstrom, A.P. (2002).Objects and events in the attentional blink. Psychol. Sci. 13,410–415.

Shiffrin, R.M., and Gardner, G.T. (1972). Visual processing capacityand attentional control. J. Exp. Psychol. 93, 72–82.

Spitzer, H., Desimone, R., and Moran, J. (1988). Increased attentionenhances both behavioral and neuronal performance. Science240, 338–340.

Tong, F., Nakayama, K., Vaughan, J.T., and Kanwisher, N. (1998).Binocular rivalry and visual awareness in human extrastriate cortex.Neuron 21, 753–759.

Treisman, A.M., and Gelade, G. (1980). A feature-integration theoryof attention. Cognit. Psychol. 12, 97–136.

Treisman, A.M., and Kanwisher, N.G. (1998). Perceiving visually pre-sented objects: recognition, awareness, and modularity. Curr. Opin.Neurobiol. 8, 218–226.

Vogel, E.K., Luck, S.J., and Shapiro, K.L. (1998). Electrophysiologicalevidence for a postperceptual locus of suppression during the atten-tional blink. J. Exp. Psychol. Hum. Percept. Perform. 24, 1656–1674.

Weiskrantz, L., Rao, A., Hodinott-Hill, I., Nobre, A.C., and Cowey,A. (2003). Brain potentials associated with conscious aftereffectsinduced by unseen stimuli in a blindsight subject. Proc. Natl. Acad.Sci. USA 100, 10500–10505.

Wojciulik, E., and Kanwisher, N. (1999). The generality of parietalinvolvement in visual attention. Neuron 23, 747–764.

Wolfe, J.M., Cave, K.R., and Franzel, S.L. (1989). Guided search: Analternative to the feature integration model for visual search. J. Exp.Psychol. Hum. Percept. Perform. 15, 419–433.

Yantis, S., Schwarzbach, J., Serences, J.T., Carlson, R.L., Steinmetz,M.A., Pekar, J.J., and Courtney, S.M. (2002). Transient neural activityin human parietal cortex during spatial attention shifts. Nat. Neu-rosci. 5, 995–1002.

27. Henson, S. A. & Warner, R. R. Male and female alternative reproductive behaviors in fishes: a new

approach using intersexual dynamics. Annu. Rev. Ecol. Syst. 28, 571–592 (1997).

28. Giraldeau, L. A. & Caraco, T. Social Foraging Theory (Princeton Univ. Press, Princeton, 2000).

29. Sober, E. & Wilson, D. S. Unto Others (Harvard Univ. Press, Cambridge, Massachusetts, 1998).

30. Boyd, R. & Richerson, P. J. Group selection among alternative evolutionarily stable strategies. J. Theor.

Biol. 145, 331–342 (1990).

Supplementary Information accompanies the paper on www.nature.com/nature.

Acknowledgements We thank A. Dornhaus, M. Enquist, E. Fehr and L.-A. Giraldeau for

comments on a previous version of this Letter.

Authors’ contributions J.M.M. formulated the main ideas as a result of conversations with A.I.H.;

J.M.M. also formulated the model, and was responsible for the material in Box 1; Z.B. carried out

the computations, and prepared the figures; A.I.H. surveyed the literature, and had the main

responsibility for writing the Letter.

Competing interests statement The authors declare that they have no competing financial

interests.

Correspondence and requests for materials should be addressed to J.M.M.

(john.mcnamara@bristol.ac.uk).

..............................................................

Neural activity predicts individualdifferences in visual workingmemory capacityEdward K. Vogel & Maro G. Machizawa

Department of Psychology, University of Oregon, Eugene, Oregon 97403-1227,USA.............................................................................................................................................................................

Contrary to our rich phenomenological visual experience, ourvisual short-term memory system can maintain representationsof only three to four objects at any given moment1,2. For over acentury, the capacity of visual memory has been shown to varysubstantially across individuals, ranging from 1.5 to about 5objects3–7. Although numerous studies have recently begun tocharacterize the neural substrates of visual memory processes8–12,a neurophysiological index of storage capacity limitations has notyet been established. Here, we provide electrophysiological evi-dence for lateralized activity in humans that reflects the encodingand maintenance of items in visual memory. The amplitude ofthis activity is strongly modulated by the number of objectsbeing held in the memory at the time, but approaches a limitasymptotically for arrays that meet or exceed storage capacity.Indeed, the precise limit is determined by each individual’smemory capacity, such that the activity from low-capacity indi-viduals reaches this plateau much sooner than that from high-capacity individuals. Consequently, this measure provides astrong neurophysiological predictor of an individual’s capacity,allowing the demonstration of a direct relationship betweenneural activity and memory capacity.

To measure the neural correlates of visual memory capacity, werecorded event-related potentials (ERPs) from normal young adultswhile they performed a visual memory task. On each trial they werepresented with a brief bilateral array of coloured squares and wereasked to remember the items in only one hemifield, which wasindicated with an arrow (Fig. 1a). Memory was tested one secondlater with the presentation of a test array that was either identical tothe memory array or differed by one colour. Subjects pressed one oftwo buttons to indicate whether the two arrays were identical ordifferent. We have used variations of this paradigm previously andhave found that observers are accurate for array sizes of up to three

to four items, and that performance is not significantly influencedby perceptual or verbal processes1,3.

In the first experiment, we recorded ERPs to the onset of a four-item memory array so that we could observe the sustained electro-physiological response during the memory retention interval. A fewprevious ERP studies have observed a sustained response duringworking memory tasks for foveally presented stimuli, but did notexamine lateralized effects13,14. In contrast, we took advantage of theprimarily contralateral organization of the visual system by pre-senting lateralized stimuli in each hemifield so that we couldmeasure the spatially specific hemispheric responses to memoryarrays that were either contralateral or ipsilateral with respect toelectrode position15,16. Approximately 200 ms after the onset of thememory array, we found a large negative-going voltage over thehemisphere that was contralateral to the memorized hemifield, andthis response persisted throughout the duration of the memoryretention interval (Fig. 1b). This response was focused primarilyover the posterior parietal and lateral occipital electrode sites andstrongly resembled delay activity recorded from individual neuronsin monkey visual cortex12,17.

Numerous processes contribute to visual memory performance,and we sought to determine which aspects of processing arereflected by the contralateral delay activity. Although this effectseems to reflect the maintenance of object representations from thememory array, it is necessary to rule out the possibility that itreflects executive processes18 involved in performing the task, oreven more general processes such as increased effort or arousal19–21.In the second experiment, we tested this by varying the number ofitems in the memory array to establish whether the amplitude issensitive to the number of representations that are being held invisual memory. Memory arrays in this experiment varied from oneto four items in each hemifield (average capacity in this task isnormally around three items3,7). To compare directly the magnitudeof activity across array sizes, we constructed ‘difference waves’ inwhich the ipsilateral activity was subtracted from the contralateralactivity for each array size, which removes the contribution of anynonspecific, bilateral ERP activity.

As shown in Fig. 2a, the amplitude was highly sensitive to thenumber of items in the memory array. Indeed, increasing an array

Figure 1 Stimuli and results from experiment one. a, Example of a visual memory trial for

the left hemifield. SOA, stimulus onset asynchrony. b, Grand averaged ERP waveforms

time-locked to the memory array averaged across the lateral occipital and posterior

parietal electrode sites in experiment one. The two grey rectangles reflect the time periods

for the memory and test arrays, respectively. Note that, by convention, negative voltage is

plotted upwards.

letters to nature

NATURE | VOL 428 | 15 APRIL 2004 | www.nature.com/nature748 © 2004 Nature Publishing Group

from one to two squares or from two to three squares resulted in asubstantial increase in amplitude. Moreover, because memoryperformance for near-capacity arrays can fluctuate over time,leading to occasional incorrect responses, we compared the ampli-tude of the delay activity for correct and incorrect trials. Theamplitude for incorrect trials was considerably smaller than thatfor correct trials (P , 0.01), further suggesting that the delayactivity specifically reflects the maintenance of successful represen-tations in visual memory. Nevertheless, it is possible that the extentof executive processes also increases with additional memory items.Moreover, there are small but reliable differences in accuracy acrossarray sizes, which leaves open the possibility that increases inarousal or effort for larger arrays may have produced the increasein amplitude.

The amplitude of the contralateral delay activity may haveincreased as the result of increasing the number of representations,more executive processing, or higher difficulty; however, thesealternatives make different predictions for array sizes that exceedvisual memory capacity. For example, when comparing a trialcontaining four memory items to a trial containing eight, thenumber of active memory representations should be approximatelyidentical, because both trials exceed a typical individual’s memorycapacity. That is, the subject can maintain only three to four itemswhether the attended side of the array contains four or eight items.In contrast, the difficulty and extent of executive processingincreases substantially for eight-item arrays compared with four-item arrays22. Indeed, this has been a significant limitation ofprevious neurophysiological studies that have reported memoryload effects, because the amount of activity continues to increase forloads that exceed capacity, indicating that these measures are notdirectly measuring memory capacity10,21,23. Therefore, in the thirdand fourth experiments, we compared the delay activity for supra-capacity arrays with memory arrays at or near capacity. If it reflectsthe active representations held in visual memory, we would expectno difference in amplitude between supra-capacity arrays and

capacity arrays. However, if it reflects executive processes or theamount of general effort, we would expect that amplitude shouldcontinue to increase for supra-capacity arrays.

The results of the third experiment show that although there wasa significant increase in amplitude from arrays of two items per sideto arrays of four items per side, there was no increase from fouritems to six items (Fig. 2b). That is, the amplitude reached a limitwith arrays of approximately four items per side. We tested thisfurther in the fourth experiment by following the same experimen-tal design but with larger array sizes (Fig. 2c). Again we found asignificant amplitude increase from two to four items per side, butno increase from four items to either eight or ten items per side.These results strongly support the hypothesis that the delay activityreflects the specific maintenance of representations in visual mem-ory because its amplitude is sensitive to the number of successfulrepresentations that are active in memory at the time. In addition,the absence of continued amplitude increase beyond capacity alsominimizes the possibility that the sub-capacity amplitude effects in

Figure 2 ERP difference waves at lateral occipital and posterior parietal electrode sites for

experiments two, three and four, respectively. a, Pairwise comparisons yielded significant

differences in amplitude between array sizes of one, two and three (P , 0.001), but no

difference between three and four items (P . 0.20) in experiment two. b, c, No

significant differences in amplitude were observed between arrays of four, six, eight or ten

items (P . 0.25 in all cases) in experiments three and four.

Figure 3 Mean amplitude and visual memory capacity. a, Mean amplitude and visual

memory capacity across experiments two, three and four. Error bars reflect 95%

confidence intervals. b, The correlation between an individual subject’s memory capacity

and the increase in amplitude of delay activity between two- and four-item arrays.

letters to nature

NATURE | VOL 428 | 15 APRIL 2004 | www.nature.com/nature 749© 2004 Nature Publishing Group

the second experiment were because of increases in the size of the‘attentional spotlight’24, because supra-capacity arrays require alarger spotlight than at- or below-capacity arrays, but show noincreases in amplitude.

The supra-capacity array sizes in these experiments providedsubstantial increases in both the extent of executive processes andthe difficulty in performing the task. For example, there was a 32%reduction in accuracy between arrays of four and ten items, yet therewas no increase in the amplitude of the contralateral delay activity.Furthermore, we also observed a more centrally distributed bilateralwave during the task that was modulated by the number of items inthe memory array. However, in sharp contrast to the contralateralactivity, the amplitude of this bilateral wave continued to increasesignificantly for arrays that exceeded memory capacity, suggestingthat it is sensitive to the amount of general effort involved inperforming the task21.

These results suggest that the contralateral delay activity indexesthe currently active representations maintained in visual memory;that is, increasing in magnitude as the number of items increases,but reaching a limit once visual memory capacity is exhausted. Todemonstrate this effect further, we quantified the mean amplitudesfor each array size for experiments two to four. As shown in Fig. 3a,amplitude increased monotonically from one to three items, butthis increase levelled off at three items. We also computed visualmemory capacity estimates for each subject, using a standardformula7,25. The mean capacity of the group was 2.8 items, whichis approximately when the memory delay activity reaches a limit.This further supports the proposal that the specific limitation invisual memory capacity determines when this delay activity reachesa limit.

To gauge this relationship more finely, we examined the varia-bility across individuals for each measure. That is, we assessedwhether a given individual’s memory capacity specifically dictateswhen his or her delay activity reaches a limit. If so, one would expectlow-capacity subjects to reach the limit for smaller array sizes thanwould high-capacity subjects. Unfortunately, it is difficult to deter-mine precisely the limit with a categorical data set such as array size(for example, there is no array size of 2.6). Instead, we computed theamplitude increase between two items and four items per side foreach subject across all experiments, the logic being that the amountof amplitude increase between these two array sizes should bespecifically determined by memory capacity. For example, when asubject with a low capacity of 1.8 items is shown a two-item array,capacity should be completely consumed with the amplitude reach-ing a limit, resulting in little or no amplitude increase from two tofour items. In contrast, a subject with a high capacity of 4.5 itemswould be expected to be well below the limit for a two-item array,and should therefore show a large increase in amplitude for a four-item array.

The magnitude of the amplitude increase between two and fouritems was plotted as a function of each subject’s memory capacity inFig. 3b. These two measures were very strongly correlated (r ¼ 0.78;P , 0.0001), with low-capacity subjects producing very little ampli-tude increase and high-capacity subjects showing larger amplitudeincreases. Importantly, an individual’s memory capacity was notsignificantly correlated with either the amplitude increase betweenarrays of four and six items or the absolute amplitude of activity fora given array size.

These results show that the observed memory delay activityindexes the maintenance of active representations in visual memory.Moreover, they demonstrate a strong neurophysiological predictorof visual memory capacity. That is, simply by measuring theamplitude increase across memory array sizes, we could accuratelypredict an individual’s memory capacity. Visual working memory isthought to have a central role within cognition because it maintainsrepresentations from the environment so that they may be acted onor manipulated3,26. Indeed, an individual’s ability to perform many

high-level cognitive functions has been shown to be directlyinfluenced by his or her memory capacity5,27–29. These resultsprovide the first link between this important cognitive limitationand neural activity. A

MethodsTwelve neurologically normal college students participated in each experiment (age rangeof 21–33) and gave informed consent according to procedures approved by the Universityof Oregon. Each of these observers performed 240 trials per condition in each experiment.All stimulus arrays were presented within two 48 £ 7.38 rectangular regions that werecentred 38 to the left and right of a central fixation cross on a grey background(8.2 cd m22). Each memory array consisted of 1–10 coloured squares (0.658 £ 0.658) ineach hemifield. Each square was selected at random from a set of seven highlydiscriminable colours (red, blue, violet, green, yellow, black and white), and a given colourcould appear no more than twice within an array. Stimulus positions were randomized oneach trial, with the constraint that the distance between squares within a hemifield was atleast 28 (centre to centre). The colour of one square in the test array was different fromthe corresponding item in the memory array in 50% of trials; the colours of the twoarrays were identical on the remaining trials. At the beginning of each trial, a centralarrow cue instructed the subjects to remember the items in either the left or the righthemifield.

We computed visual memory capacity using a formula developed by Pashler23 andrefined by Cowan7. Essentially, this approach assumes that if an observer can hold K itemsin memory from an array of S items, then the item that changed should be one of the itemsbeing held in memory on K/S trials, leading to correct performance on K/S of the trials onwhich an item changed. To correct for guessing, this procedure also takes into account thefalse alarm rate. The formula is K ¼ S £ (H 2 F), where K is the memory capacity, S is thesize of the array, H is the observed hit rate and F is the false alarm rate.

ERPs were recorded in each experiment using our standard recording and analysisprocedures30, including rejection of trials contaminated by blinks or large (.18) eyemovements. We recorded from 22 standard electrode sites (international 10/20 system)spanning the scalp. We computed contralateral waveforms by averaging the activityrecorded at right hemisphere electrode sites when subjects were cued to remember the leftside of the memory array with the activity recorded from the left hemisphere electrodesites when they were cued to remember the right side. Contralateral delay activity wasmeasured at posterior parietal, lateral occipital and posterior temporal electrode sites asthe difference in mean amplitude between the ipsilateral and contralateral waveforms,with a measurement window of 300–900 ms after the onset of the memory array.

Received 23 December 2003; accepted 26 February 2004; doi:10.1038/nature02447.

1. Luck, S. J. & Vogel, E. K. The capacity of visual working memory for features and conjunctions. Nature

390, 279–281 (1997).

2. Sperling, G. The information available in brief visual presentations. Psychol. Monogr. 74, Whole No.

498 (1960).

3. Vogel, E. K., Woodman, G. F. & Luck, S. J. Storage of features, conjunctions, and objects in visual

working memory. J. Exp. Psychol. Hum. Percept. Perform. 27, 92–114 (2001).

4. Jevons, W. S. The power of numerical discrimination. Nature 3, 281–282 (1871).

5. Engle, R. W., Kane, M. J. & Tuholski, S. W. in Models of Working Memory: Mechanisms of Active

Maintenance and Executive Control (eds Miyake, A. & Shah, P.) 102–134 (Cambridge Univ. Press, New

York, 1999).

6. Miller, G. A. The magical number seven, plus or minus two: Some limits on our capacity for

processing information. Psychol. Rev. 63, 81–97 (1956).

7. Cowan, N. The magical number 4 in short-term memory: A reconsideration of mental storage

capacity. Behav. Brain Sci. 24, 87–185 (2001).

8. Jonides, J. et al. Spatial working memory in humans as revealed by PET. Nature 363, 623–625 (1993).

9. Courtney, S. M., Ungerleider, L. G., Keil, K. & Haxby, J. V. Transient and sustained activity in a

distributed neural system for human working memory. Nature 386, 608–611 (1997).

10. Cohen, J. D. et al. Temporal dynamics of brain activation during a working memory task. Nature 386,

604–608 (1997).

11. Miller, E. K., Erickson, C. A. & Desimone, R. Neural mechanisms of visual working memory in

prefrontal cortex of the macaque. J. Neurosci. 16, 5154–5167 (1996).

12. Fuster, J. M. & Jervey, J. P. Neuronal firing in the inferotemporal cortex of the monkey in a visual

memory task. J. Neurosci. 2, 361–375 (1982).

13. Ruchkin, D., Johnson, R., Grafman, J., Canoune, H. & Ritter, W. Multiple visuospatial working

memory buffers: Evidence from spatiotemporal patterns of brain activity. Neuropsychologia 35,

195–209 (1997).

14. Ruchkin, D., Johnson, R., Canoune, H. & Ritter, W. Short-term memory storage and retention: An

event-related brain potential study. Electroencephalogr. Clin. Neurophysiol. 76, 419–439 (1990).

15. Gratton, G. The contralateral organization of visual processing: A theoretical concept and a research

tool. Psychophysiology 35, 638–647 (1998).

16. Woodman, G. F. & Luck, S. J. Electrophysiological measurement of rapid shifts of attention during

visual search. Nature 400, 867–869 (1999).

17. Miller, E. K., Li, L. & Desimone, R. Activity of neurons in anterior inferior temporal cortex during a

short-term memory task. J. Neurosci. 13, 1460–1478 (1993).

18. Baddeley, A. Exploring the central executive. Q. J. Exp. Psychol. A 49, 5–28 (1996).

19. Hillyard, S. A., Vogel, E. K. & Luck, S. J. Sensory gain control (amplification) as a mechanism of

selective attention: Electrophysiological and neuroimaging evidence. Phil. Trans. R. Soc. Lond. B 353,

1257–1270 (1998).

20. Rypma, B. & D’Esposito, M. D. The roles of prefrontal brain regions in components of working

memory. Proc. Natl Acad. Sci. USA 96, 6558–6563 (1999).

21. Ruchkin, D., Canoune, H., Johnson, R. & Ritter, W. Working memory and preparation elicit different

patterns of slow wave event-related brain potentials. Psychophysiology 32, 399–410 (1995).

letters to nature

NATURE | VOL 428 | 15 APRIL 2004 | www.nature.com/nature750 © 2004 Nature Publishing Group

22. Rypma, B. & D’Esposito, M. D. A subsequent-memory effect in dorsolateral prefrontal cortex. Cogn.

Brain Res. 16, 162–166 (2003).

23. Rypma, B., Prabhakaran, V., Desmond, J., Glover, G. H. & Gabrieli, J. D. Load-dependent roles of

frontal brain regions in the maintenance of working memory. Neuroimage 9, 216–226 (1999).

24. Eriksen, C. W. & St James, J. D. Visual attention within and around the field of focal attention: A zoom

lens model. Percept. Psychophys. 40, 225–240 (1986).

25. Pashler, H. Familiarity and visual change detection. Percept. Psychophys. 44, 369–378 (1988).

26. Logie, R. H. Visuo-Spatial Working Memory (Erlbaum, Hove, UK, 1995).

27. Kane, M. J. & Engle, R. W. Working memory capacity and the control of attention: The contributions

of goal neglect, response competition, and task set to Stroop interference. J. Exp. Psychol. Gen. 132,

47–70 (2003).

28. Daneman, M. & Merikle, P. M. Working memory and language comprehension: A meta-analysis.

Psychon. Bull. Rev. 3, 422–433 (1996).

29. Kyllonen, P. C. & Christal, R. E. Reasoning ability is (little more than) working memory capacity.

Intelligence 14, 398–433 (1990).

30. Vogel, E. K., Luck, S. J. & Shapiro, K. L. Electrophysiological evidence for a postperceptual locus of

suppression during the attentional blink. J. Exp. Psychol. Hum. Percept. Perform. 24, 1656–1674

(1998).

Acknowledgements The research reported here was supported by a grant from the US National

Institute of Mental Health.

Competing interests statement The authors declare that they have no competing financial

interests.

Correspondence and requests for materials should be addressed to E.K.V.

(vogel@darkwing.uoregon.edu).

..............................................................

Capacity limit of visualshort-term memory inhuman posterior parietal cortexJ. Jay Todd & Rene Marois

Vanderbilt Vision Research Center, Department of Psychology, VanderbiltUniversity, 530 Wilson Hall, Nashville, Tennessee 37203, USA.............................................................................................................................................................................

At any instant, our visual system allows us to perceive a rich anddetailed visual world. Yet our internal, explicit representation ofthis visual world is extremely sparse: we can only hold in mind aminute fraction of the visual scene1,2. These mental represen-tations are stored in visual short-term memory (VSTM). Eventhough VSTM is essential for the execution of a wide array ofperceptual and cognitive functions3–5, and is supported by anextensive network of brain regions6–9, its storage capacity isseverely limited10–13. With the use of functional magnetic reso-nance imaging, we show here that this capacity limit is neurallyreflected in one node of this network: activity in the posteriorparietal cortex is tightly correlated with the limited amount ofscene information that can be stored in VSTM. These resultssuggest that the posterior parietal cortex is a key neural locus ofour impoverished mental representation of the visual world.

To investigate the neural basis of VSTM’s storage capacity limit,17 subjects were scanned while performing a parametric loadmanipulation14 of a delayed visual matching-to-sample task(Fig. 1). On each trial, subjects were briefly presented with a sampledisplay containing one to eight coloured discs and, after a 1,200-msretention interval, decided whether a single probe disc matchedone of the sample discs in location and colour. A 1,200-ms delaymaximizes VSTM’s capacity: with delays shorter than 1 s, VSTMcapacity is inflated by sensory (iconic) representations of the dis-play15, whereas long delays not only underestimate VSTM capacityowing to memory degradation15, but also favour the recruitment ofrehearsal mechanisms and verbal/abstract recoding of the visualmaterial16. To minimize verbal strategies further, a verbal working-

memory/articulatory suppression task was administered concur-rently with the VSTM task: throughout the trial, subjects rehearsedtwo digits presented at trial onset and reported them at trial offset.Performance in this task was high and independent of VSTM set size(92–94% accuracy across set sizes; F 5,80 ¼ 0.64, P ¼ 0.67), attestingto the absence of a trade-off between the verbal and visual tasks, aspredicted from the independence of these two working-memorysystems17,18.

Accuracy in the VSTM task declined with increased set size (setsize 1, 97.7%; set size 2, 94.2%; set size 3, 90.0%; set size 4, 86.2%; setsize 6, 73.3%; set size 8, 68.5%). The number of objects encoded ateach set size, estimated with Cowan’s K formula11, increased up toset size 3 or 4, and levelled off thereafter (Fig. 2; t-test between setsizes 4 and 8, P . 0.05). This behavioural function is fittedsignificantly better by a quadratic function than by a linear function(P ¼ 0.01)19. Thus, VSTM storage capacity is about three or fouritems, which is consistent with previous studies11,13. Importantly,this capacity limit is not due to insufficient time to encode items inVSTM4. Tripling the sample presentation time from 150 to 450 msin a separate experiment did not affect the K function (n ¼ 16,P ¼ 0.28), an observation consistent with previous findings12,13. TheVSTM task therefore expresses the capacity limit of VSTM storageas opposed to a limitation in spatially attending to the display orencoding items in VSTM.

The brain substrates mediating VSTM’s storage capacity limitshould demonstrate a response profile paralleling the behavioural Kfunction: activation should increase until set size 3 or 4 and level offthereafter. To isolate such regions, a voxel-based multiple regressionanalysis with K-weighted set size coefficients was performed. Theresulting statistical parametric maps revealed a single bilaterallysymmetric area in the intraparietal and intraoccipital sulci (IPS/IOS; P , 0.05 corrected). Time-course analysis (Fig. 3a) confirmeda strong correlation between the IPS/IOS peak response amplitudeand the number of objects encoded (r ¼ 0.54, P , 0.001; Fig. 2).The peak blood oxygenation level-dependent (BOLD) responsefunction reached a plateau by set size 4 (t-test between 4 and 8,P , 0.05) and was better described by a quadratic function than bya linear function (P , 0.01). This parietal activation is not simplyrelated to task difficulty: accuracy decreased and reaction

Figure 1 Trial design. Each trial began with the auditory presentation of two digits to be

rehearsed throughout the trial. A sample display containing a variable number of

coloured discs was then presented for 150 ms, followed by a 1,200-ms retention period,

and then by a single coloured probe disc. Subjects judged whether the colour of the probe

matched the colour of the disc shown at the same position in the sample display.

Afterwards, two digits appeared and subjects indicated whether these were the same as

those presented at trial onset.

letters to nature

NATURE | VOL 428 | 15 APRIL 2004 | www.nature.com/nature 751© 2004 Nature Publishing Group

Switching Memory Systems during Learning: Changes inPatterns of Brain Acetylcholine Release in the Hippocampusand Striatum in Rats

Qing Chang and Paul E. GoldDepartment of Psychology, University of Illinois at Urbana-Champaign, Champaign, Illinois 61820

This experiment measured acetylcholine (ACh) release simultaneously in the hippocampus and striatum while rats were trained in across maze. Consistent with past findings, rats initially showed learning on the basis of place (i.e., turning to the correct position relativeto the room), but after extensive training, rats shifted to learning on the basis of response (i.e., turning to the right/left to find the food).Profiles of ACh release in the hippocampus and striatum were markedly different during training. In the hippocampus, ACh releaseincreased by �60% at the onset of training and remained at that level of release throughout training, even after the rats began to showlearning on the basis of turning rather than place. In the striatum, increases in ACh release occurred later, reaching asymptotic increasesof 30 – 40%, coincident with a transition from expressing place learning to expressing response learning. These findings suggest that thehippocampal and striatal systems both participate in learning in this task, but in a manner characterized by differential activation of theneural systems. The hippocampal system is apparently engaged first before the striatum is activated and, to the extent the hippocampusis important for place learning, promotes the use of a place solution to the maze. Later in training, although the hippocampus remainsactivated, the striatum is also activated in a manner that may enable the use of a response strategy to solve the maze. These findings mayoffer a neurobiological marker of a transition during skill learning from declarative to procedural learning.

Key words: hippocampus; striatum; learning strategies; interactions between memory systems; acetylcholine and regulation of memory;spatial versus response learning systems

IntroductionDifferent classes of memory appear to be processed by relativelydistinct memory systems (Cohen and Squire, 1980; Gabrieli,1998; Kesner, 1998; Willingham, 1998; Gold et al., 2001; Kim andBaxter, 2001; Packard, 2001; White and McDonald, 2002). Theprincipal evidence comes from studies of multiple dissociationsof brain structure and memory functions. However, there are alsodemonstrations showing that damage to one system can enhancelearning mediated by another system (Matthews and Best, 1995;McDonald and White, 1995; Matthews et al., 1999; White andWallet, 2000; Ferbinteanu and McDonald, 2001). These findingssupport the view that multiple memory systems at times competewith each other for control over learning.

Most studies of multiple memory systems use tasks that aredependent on one but not another neural system. One task inwhich the contributions of two systems can be contrasted is across maze. Rats are trained in a T-configuration of the maze, inwhich they are trained, for example, to go from the south arm tothe east arm for reward. This simple task requirement can besolved using either response (egocentric) or place (allocentric)mechanisms. This task has a long history (Tolman et al., 1946,1947) with detailed analyses of the impact of many variables on

learning (Restle, 1957). The solution used by each rat is revealedon a probe trial on which the rat begins from an arm 180 o (e.g.,north) from the original start arm. The cross maze can be used toaddress the issue of whether rats preferentially use learning on thebasis of habit (i.e., turn right to approach the goal) or on the basisof place (i.e., go to a particular part of the room to approach thegoal). Recent evidence indicates quite clearly that both types oflearning contribute to acquisition of this task. In particular, ratsgenerally use place solutions early in training and response solu-tions later in training (Packard and McGaugh, 1996). Lidocaine-induced inactivation of the hippocampus or striatum decreasesthe expression of place and response solutions, respectively(Packard and McGaugh, 1996). Conversely, glutamate-inducedactivation of the hippocampal and striatal systems increases theexpression of place and response solutions, respectively (Packard,1999). Additionally, profiles of acetylcholine (ACh) release in thehippocampus and striatum, evident even before initial exposureto the maze, predict the preferred solution at the time rats reachcriterion performance (McIntyre et al., 2003a). However, thispredilection to use place or response learning does not provide afull explanation of the behavior because, after extensive training,most rats eventually select response strategies at the time of aprobe trial (Packard and McGaugh, 1996; Packard, 1999). There-fore, rats switch from place to response [or from cognitive tohabit (Packard, 1999)] solutions with extensive training. The is-sue for the present report is whether the switch in strategy used tosolve the maze is reflected in changes in the profiles of ACh re-lease in the hippocampus and striatum, used in this study to markchanges in the activation of these neural systems.

Received Aug. 12, 2002; revised Oct. 10, 2002; accepted Dec. 27, 2002.This work was supported by United States Public Health Service research grants from the National Institute on

Aging (AG 07648) and the National Institute of Neurological Disorders and Stroke (NS 32914), by the United StatesDepartment of Agriculture (00-35200-9839), and by the Alzheimer’s Association.

Correspondence should be addressed to Dr. Paul E. Gold, Department of Psychology, University of Illinois atUrbana-Champaign, 603 East Daniel Street, Champaign, IL 61820. E-mail: pgold@uiuc.edu.Copyright © 2003 Society for Neuroscience 0270-6474/03/233001-05$15.00/0

The Journal of Neuroscience, April 1, 2003 • 23(7):3001–3005 • 3001

Materials and MethodsSubjects. Six male Sprague Dawley rats (Hilltop Laboratories, Scottdale,PA), weighing 380 – 430 gm at the time of surgery, were used as subjects.The rats were housed individually in translucent cages, with food andwater available ad libitum until food restriction began. The rats weremaintained in a 12 hr light/dark cycle (lights on at 7:00 A.M.) throughoutthe experiment.

Surgery. Each rat was anesthetized with sodium pentobarbital (50 – 60mg/kg, i.p.). The rats were placed in a stereotaxic apparatus with hori-zontal skull (Paxinos and Watson, 1986). Two plastic guide cannulas (3mm, CMA/12; Carnegie Medicin, Stockholm, Sweden) were loweredinto the hippocampus (coordinates, 5.0 mm posterior to bregma, 5.0mm lateral and 4.2 mm ventral from the surface of the skull) and thelateral striatum (0.3 mm posterior to bregma, 4.3 mm lateral and 3.7 mmventral from the surface of the skull). The use of ventral (vs dorsal)hippocampal placement was based on evidence showing that, at least interms of amygdala-dependent tasks, the ventral hippocampus appears tobe important for competition during learning (Ferbinteanu and Mc-Donald, 2001). Recent findings obtained with measures of ACh release inthe ventral hippocampus are consistent with this view (McIntyre et al.,2002).

One-half of the subjects had cannulas in the left hippocampus and theright striatum, and one-half had the converse implantation. The guidecannulas were anchored in place with dental cement and skull screws.Stylets made to be flush with the cannula tips were inserted into thecannulas until the start of microdialysis procedures.

Behavioral procedures. Approximately 1 week after surgery, the ratswere placed on a food-restriction schedule such that their body weightswere gradually reduced to and maintained at 80%. During this 7–10 dperiod, the animals were weighed and handled (5 min/d) every day.

The training apparatus was an elevated plus-shaped cross maze with ablack Plexiglas floor and clear Plexiglas walls. The maze had four identicalarms (length � width � height, 45 � 12.5 � 15 cm) containing foodwells at their ends; the arms extended from a central platform (12.5 �12.5 � 15 cm). Training was conducted in a lighted training room con-taining moderate-density extramaze cues (e.g., posters on the walls, door,and polygraph in a corner of the room).

On training trials, the arm facing the start arm was always blocked,creating a T-maze. Rats were trained to turn right to obtain a half piece ofFrosted Cheerio (General Mills, Minneapolis, MN) located at the end ofone arm. Thus, each rat could use either an allocentric spatial strategy (goto the arm in a fixed position of the room), or egocentric nonspatialstrategy (turn right) to obtain the food reward.

On each trial, a rat was placed at the end of start arm, facing the centerof the maze, and given up to 45 sec to find and to eat the food. After thefood was eaten or the time expired, the rat was placed in a holding cagefor an interval such that the time from the start of one trial to the next was60 sec. This was done to synchronize training with collection of micro-dialysis samples so that a 5 min microdialysis sample represented fivetrials. To avoid possible unintended influences of intramaze cues, themaze was turned 90 o clockwise on every trial.

A probe test was administered after each block of 20 trials. On theprobe trial, the cross maze was configured as a T-maze with the start arm180 o from the original start arm. Both arms at the ends of the T-mazewere baited on the probe trial. If, on the probe trial, the rat turned in thedirection that was correct during training, the behavior was termed aresponse solution. If the rat turned instead toward the arm now locatedin the correct place in the room, the behavior was termed a place solution.

Probe trials were administered near the middle of a 5 min intervalbetween trials 20 and 21, 40 and 41, 60 and 61, 80 and 81, and after 100.Except for the time spent on the probe trial, each rat was kept in theholding cage for the remainder of the 5 min. Thus, for each rat, there werefive training blocks of 20 trials each (total � 100 trials) and five probetrials. Training was completed on a single day within a single session.

Microdialysis procedures. One microdialysis probe was insertedthrough a guide cannula into the hippocampus, and another probe wasinserted into the contralateral striatum. The dialysis probes were per-fused continuously at a rate of 2.0 �l/min with artificial CSF (in mM: 128

NaCl, 2.5 KCl, 1.3 CaCl2, 2.1 MgCl2, 0.9 NaH2PO4, 2.0 Na2HPO4, 1.0dextrose, adjusted to pH 7.4) that contained a 100 nM concentration ofthe acetylcholinesterase inhibitor neostigmine.

To allow equilibration with brain extracellular fluid and to avoid tem-porary changes in extracellular neurotransmitter levels caused by acutetissue damage (Westerink and Timmerman, 1999), the first hour of dia-lysate was discarded. During this hour, the animals were kept in a holdingcage with fresh bedding. Next, dialysate samples were collected every 5min into small vials by an automatic refrigerated fraction collector(CMA/170; Carnegie Medicin). The first four samples were collectedwhile rats remained undisturbed in the holding cage and comprised thesamples for baseline values. After the baseline sampling period, rats weretrained while samples continued to be collected every 5 min. At theconclusion of behavioral testing, samples were sealed and stored at�20°C until assay for ACh content.

After in vivo sampling was completed, each microdialysis probe wasremoved from the rat and placed into a 100 nM ACh standard solution for10 –15 min to determine relative recovery.

ACh assay procedures. ACh content in each dialysate sample was as-sayed by HPLC in combination with an electrochemical detector (BAS;Bioanalytical Systems, West Lafayette, IN). The assay system included anion-exchange microbore analytical column (BAS P/N MF-8904, 530 � 1mm), a microbore ACh/choline immobilized enzyme reactor containingacetylcholinesterase and choline oxidase (BAS P/N MF-8903, 50 � 1mm), an auxiliary electrode with radical flow electrochemical thin-layercell and 13 mm thin-layer gasket, a wired enzyme electrode kit (a redoxpolymer film containing horseradish peroxidase coated on the surface ofa 3 mm glassy carbon working electrode), and a low-dispersion injectionvalve with a 5 �l polyetheretherketone loop (Rheodyne model 9125-087). Stable and relatively pulse-free flow was achieved with a Shimadzu(Tokyo, Japan) LC-10ADvp pump with microstep plunger.

The potential held by the working electrode was 100 mV versus anAg/AgCl reference electrode. The mobile phase contained 50 mM

Na2HPO4, pH 8.5, and 0.5% Kathon (BAS P/N CF-2150). The flow ratewas 140 �l/min. Injection volume in this experiment was 2.5 �l. Thedetection limit was �5 fmol. The assay was completed within 13 min.

Histology. After training was completed, rats were deeply anesthetizedwith sodium pentobarbital and perfused with saline followed by 10%formalin solution. The brains were removed and placed in 10% formalinsolution for 1–2 weeks before sectioning. Each brain was then frozen(�20°C) and sliced (50 �m) with a Leica (Nussloch, Germany) 1800cryostat. Sections through probe sites were mounted on slides, dried, andstained with cresyl violet. Figure 1 illustrates acceptable placement loca-tions for the probes.

ResultsAs shown in Figure 2, the rats learned to approach the correct armquite rapidly. Accuracy improved from 50% during the first 10trials to 87% in the second 10 trials, and then reached and stayedat �95% throughout the rest of training. The mean number oftrials taken before beginning a run of 9 of 10 correct choices was20.2 � 2.3.

The performance observed on the probe trials administeredafter each 20 trials showed a steady change from place to responsestrategies (Fig. 3). The first probe trial occurred when mean per-formance was between 87 and 95% correct (Fig. 2). On this probetrial, five of the six rats made place selections. With continuedtraining through and beyond criterion performance, the ratschanged their selection on the probe trials until, after 100 trials,all six rats made response selections (probe 1 vs 6; � 2 � 8.571; p �0.005).

A key issue in this experiment was whether the profile ofchanges in ACh release during training was different in the hip-pocampus and striatum. Concentrations of ACh in baseline sam-ples were 7.3 � 2.1 and 37.3 � 12.3 fmol/�l for the hippocampusand striatum, respectively. As shown in Figure 4, release of ACh inthe hippocampus increased by �60% immediately on the start of

3002 • J. Neurosci., April 1, 2003 • 23(7):3001–3005 Chang and Gold • Switching Memory Systems

training and remained at that level throughout the training trials.In marked contrast, ACh release in the striatum increased muchmore slowly, reaching its asymptotic increase of 30 – 40% aftertrial 25. For both hippocampal and striatal samples, the magni-tudes of ACh release during the 5 min samples taken at the time ofthe probe trials were not different from those seen on the trialsimmediately before and after and are therefore not shown here.

Of particular interest, the different times at which release ofACh increased in the hippocampus versus striatum corre-sponded to the changes in the preferred solution displayed by ratson the probe trials. Early in training, when the rats exhibited aplace strategy on probe trials, release of ACh had already in-

creased to its asymptote in the hippocampus but was still low instriatum. Later in training, when the rats switched to the use of aresponse strategy on the probe trial, release of ACh in the stria-tum had also increased. Importantly, ACh release in the hip-pocampus did not decline late in training but rather remainedhigh throughout training.

In past work from our laboratory, the ratio of baseline AChrelease in the hippocampus versus striatum predicted the perfor-mance on a single probe test given immediately after rats reachedcriterion performance (McIntyre et al., 2003a). In the presentexperiment, the corresponding probe trial could come between 1and 20 trials after the criterion was reached. Although an identicalmeasure was not available in the present experiment, it was pos-sible to examine the relationship between baseline release of AChand the probe test number at which the subject first used a re-sponse strategy. As shown in Figure 5, the ratio of baseline releaseof ACh in the hippocampus/striatum was significantly correlatedwith the number of trials to the first selection of a response strat-egy on a probe trial (r � 0.83; p � 0.05). Those rats with thelowest ratio of ACh release in the hippocampus versus striatumwere the first to switch from a place to a response strategy, andthose with the highest ratio of ACh release in the hippocampusversus striatum were the last to switch to a response strategy.

Figure 1. Examples of acceptable placements of microdialysis probes in the hippocampus( A) and striatum ( B). [Adapted from Paxinos and Watson (1986).]

Figure 2. Rate of acquisition on cross maze. All rats reached 9 of 10 correct choices by trial 30regardless of strategy shown on nearest probe trials.

Figure 3. Transition from place strategy to response strategy during 100 training trials. Aprobe trial was administered after every 20 training trials.

Figure 4. Changes in ACh release across 100 training trials. Each sample represents 5 minbefore and during training. During training, the samples correspond to five trials per sample.ACh release in the hippocampus increased at the onset of training and remained stable. Incontrast, ACh release in the striatum grew gradually over trials as the rats switched from place toresponse strategy.

Chang and Gold • Switching Memory Systems J. Neurosci., April 1, 2003 • 23(7):3001–3005 • 3003

DiscussionThe main finding of this experiment is that the pattern of changesin ACh release in the hippocampus and striatum during learningprovides a neurochemical marker of differential activation ofthese systems in a manner associated with differential expressionof learned responses for a task that has two effective solutions.Rats quickly learned to enter one arm of the maze to obtain food.However, consistent with previous findings (Packard and Mc-Gaugh, 1996; Packard, 1999), the basis by which the rats selectedthe correct arm changed from place to response attributes afterextended training. The present results provide evidence that in-creases in ACh release in the hippocampus precede increases inACh release in the striatum during training on a task in whichlearning proceeds sequentially from place to response strategies.These findings provide the first neurobiological measures ob-tained during training marking the transition in the memorysystems controlling learned performance during training.

The present findings used release of ACh to monitor the ex-tent and timing of participation of different brain regions duringlearning. ACh release in the hippocampus increased to its maxi-mum extent on the first trials and remained at that maximumthroughout training, suggesting that the hippocampus was en-gaged at the outset of learning and remained engaged throughoutextended training. In contrast, ACh release in the striatum in-creased more gradually, with the increase appearing as rats madetheir transition from selecting place solutions to selecting re-sponse solutions on probe trials. Thus, it appears that the hip-pocampus was activated before the striatum (i.e., at the time ratsused place solutions to solve the maze). The striatal system wasactivated later, at the time rats began to use response solutions.

Furthermore, the present findings suggest that the hippocam-pus remained activated, but that the use of hippocampal process-ing was overridden when the striatum became fully engaged. Thisinterpretation is consistent with the finding that inactivation ofthe striatum late in training leads not to chance performance butto correct performance on the basis of place learning (Packardand McGaugh, 1996). According to this view, the hippocampusretains the capacity to provide a place solution to the task and, interms of ACh release as a marker, remains activated during per-formance of the task. However, this capacity to provide a placesolution is obscured late in training by striatal contributions,unless the later striatal contribution is diminished or removed bypharmacological manipulation as in Packard and McGaugh

(1996). Importantly, on the basis of both measures of ACh releaseand manipulations of the striatum, hippocampal participationduring learning does not appear to wane even after extensivetraining. It is intriguing to consider the possibility that the se-quential activation of the hippocampal and striatal systems mayreflect a transition from declarative to procedural memories.

Confirming previous findings (McIntyre et al., 2003a), therelative release of ACh in the hippocampus and striatum assessedbefore the beginning of training predicts which individual ratswill switch from place to response solutions early and late intraining. This aspect of the results suggests that individual differ-ences in the profiles of ACh release across neural systems reflectthe extent of bias between competing memory systems duringlearning. The present findings indicate that the bias, as well asrelative release of ACh in the hippocampus and striatum, canchange during training.

In many circumstances, training-related increases in release ofacetylcholine in different neural systems provide a marker of thedegree of activation of those systems during learning (Ragozzinoet al., 1994, 1996, 1998; Fadda et al., 1996, 2000; Orsetti et al.,1996; Stancampiano et al., 1999; Nail-Boucherie et al., 2000; Goldet al., 2001; Hironaka et al., 2001; McIntyre et al., 2002, 2003a,b);findings within and across these studies show that locomotoractivity per se does not define well the conditions when increasesin ACh release are evident. These findings, like those of thepresent study, do not deal directly with the issue of how AChrelease regulates the participation of different neural systems inlearning and memory. More generally, the findings do not deter-mine whether ACh release is an initiator or a consequence ofactivation. At a general level, the findings are consistent both withviews applying modulation of memory formation to a systems-level understanding of memory processing (Gold et al., 2001) andwith views considering ACh to be a regulator of attention (Everittand Robbins, 1997; Wenk, 1997). Also, although there are find-ings suggesting that lesions of forebrain cholinergic systems havelittle or no impact on learning and memory (Baxter et al., 1996;Perry et al., 2001), these findings may in part reflect limited dam-age to cholinergic functions (Wrenn and Wiley, 1998; Gutierrezet al., 1999; Miranda and Bermudez-Rattoni, 1999).

At a mechanistic level, the seemingly analogous actions ofACh in the hippocampus and striatum are somewhat surprisinggiven the major differences in neuroanatomical organization ofthe cholinergic systems in these brain areas. Although hippocam-pal ACh is derived entirely from diagonal band/medial septumprojection neurons, striatal ACh is primarily derived from intrin-sic cholinergic neurons (Woolf and Butcher, 1981; Woolf, 1991;Calabresi et al., 2000). However, in both instances, it may beimportant that the axonal fields for these cells are very extensivein both the hippocampus and striatum, keeping open the possi-bility of similar neurobiological functions for ACh in both neuralsystems. For example, the extensive distribution of terminal fieldsseems consistent with a modulatory function for ACh released inboth neural systems studied here, perhaps upregulating local pro-cessing important for these systems to contribute to performanceon this task.

Using a variety of techniques, many experiments have shownthat multiple memory systems are responsible for learning in therat (McDonald and White, 1995; Gold et al., 2001; Kim and Bax-ter, 2001; Packard, 2001). The nature by which these systemsinteract is still under active exploration. The interactions at timesappear to be competitive in nature; for example, the removal ofthe hippocampus appears to enhance learning mediated by othersystems (Matthews and Best, 1995; McDonald and White, 1995;

Figure 5. Relationship between the ratio of baseline (pretraining) release of ACh in thehippocampus and striatum and the probe on which the rat first switched from a place to aresponse solution to the maze.

3004 • J. Neurosci., April 1, 2003 • 23(7):3001–3005 Chang and Gold • Switching Memory Systems

Matthews et al., 1999; White and Wallet, 2000; Ferbinteanu andMcDonald, 2001). In past experiments using ACh release toapproach this issue, there is evidence that the magnitude of AChrelease in the hippocampus is inversely related to acquisitionof a task dependent on the amygdala (McIntyre et al., 2002).However, competition is not the only form evident for interac-tions between memory systems. ACh release in the amygdala ispositively correlated with performance on a hippocampus-dependent task (McIntyre et al., 2003b). These findings suggestthat the hippocampus and the amygdala have a nonreciprocalinteraction in which the hippocampus competes with the amyg-dala but the amygdala cooperates with the hippocampus duringlearning. The present findings add significantly to an under-standing of the relationships between neural systems importantfor processing different classes of learning by adding both time(the stage of training) and shifts in the relative extent of AChrelease in different memory systems as features important in un-derstanding the coordination of multiple memory systems inproducing learned behavior.

ReferencesBaxter MG, Bucci DJ, Sobel TJ, Williams MJ, Gorman LK, Gallagher M

(1996) Intact spatial learning following lesions of basal forebrain cholin-ergic neurons. NeuroReport 7:1417–1420.

Calabresi P, Centonze D, Gubellini P, Pisani A, Bernardi G (2000)Acetylcholine-mediated modulation of striatal function. Trends Neurosci23:120 –126.

Cohen NJ, Squire LR (1980) Preserved learning and retention of pattern-analyzing skill in amnesia: dissociation of knowing how and knowingthat. Science 210:207–210.

Everitt BJ, Robbins TW (1997) Central cholinergic systems and cognition.Annu Rev Psychol 48:649 – 684.

Fadda F, Melis F, Stancampiano R (1996) Increased hippocampal acetylcholinerelease during a working memory task. Eur J Pharmacol 307:R1–R2.

Fadda F, Cocco S, Stancampiano R (2000) Hippocampal acetylcholine re-lease correlates with spatial learning performance in freely moving rats.NeuroReport 11:2265–2269.

Ferbinteanu J, McDonald RJ (2001) Dorsal/ventral hippocampus, fornix,and conditioned place preference. Hippocampus 11:187–200.

Gabrieli JDE (1998) Cognitive neuroscience of human memory. Annu RevPsychol 49:87–115.

Gold PE, McIntyre C, McNay E, Stefani MR, Korol DL (2001) Neurochem-ical referees of dueling memory systems. In: Memory consolidation: es-says in honor of James L. McGaugh—a time to remember (Gold PE,Greenough WT, eds), pp 219 –248. Washington, DC: American Psycho-logical Association.

Gutierrez H, Gutierrez R, Silva-Gandarias R, Estrada J, Miranda MI,Bermudez-Rattoni F (1999) Differential effects of 192IgG-saporin andNMDA-induced lesions into the basal forebrain on cholinergic activityand taste aversion memory formation. Brain Res 834:136 –141.

Hironaka N, Tanaka K, Izaki Y, Hori K, Nomura M (2001) Memory-relatedacetylcholine efflux from rat prefrontal cortex and hippocampus: a mi-crodialysis study. Brain Res 901:143–150.

Kesner RP (1998) Neurobiological views of memory. In: Neurobiology oflearning and memory (Martinez Jr JL, Kesner RP, eds), pp 361– 416. SanDiego: Academic.

Kim JJ, Baxter MG (2001) Multiple brain-memory systems: the whole doesnot equal the sum of its parts. Trends Neurosci 24:324 –330.

Matthews DB, Best PJ (1995) Fimbria/fornix lesions facilitate the learning ofnonspatial response task. Psychon Bull Rev 2:113–116.

Matthews DB, Ilgen M, White AM, Best PJ (1999) Acute ethanol adminis-tration impairs spatial performance while facilitating nonspatial perfor-mance in rats. Neurobiol Learn Mem 72:169 –179.

McDonald RJ, White NM (1995) Information acquired by the hippocam-pus interferes with acquisition of the amygdala-based conditioned-cuepreference in the rat. Hippocampus 5:189 –197.

McIntyre CK, Pal SN, Marriott LK, Gold PE (2002) Competition betweenmemory systems: acetylcholine release in the hippocampus correlates nega-

tively with good performance on an amygdala-dependent task. J Neurosci22:1171–1176.

McIntyre CK, Marriott LK, Gold PE (2003a) Patterns of brain acetylcholinerelease predict individual differences in preferred learning strategies inrats. Neurobiol Learn Mem, in press.

McIntyre CK, Marriott LK, Gold PE (2003b) Cooperation between memorysystems: acetylcholine release in the amygdala correlates positively withperformance on a hippocampus-dependent task. Behav Neurosci, inpress.

Miranda MI, Bermudez-Rattoni F (1999) Reversible inactivation of the nu-cleus basalis magnocellularis induces disruption of cortical acetylcholinerelease and acquisition, but not retrieval, of aversive memories. Proc NatlAcad Sci USA 96:6478 – 6482.

Nail-Boucherie K, Dourmap N, Jaffard R, Costentin J (2000) Contextualfear conditioning is associated with an increase of acetylcholine release inthe hippocampus of rat. Brain Res Cogn Brain Res 9:193–197.

Orsetti M, Casamenti F, Pepeu G (1996) Enhanced acetylcholine release inthe hippocampus and cortex during acquisition of an operant behavior.Brain Res 724:89 –96.

Packard MG (1999) Glutamate infused posttraining into the hippocampusor caudate-putamen differentially strengthens place and response learn-ing. Proc Natl Acad Sci USA 96:12881–12886.

Packard MG (2001) Amygdala modulation of multiple memory systems. In:Memory consolidation: essays in honor of James L. McGaugh—a time toremember (Gold PE, Greenough WT, eds), pp 201–218. Washington,DC: American Psychological Association.

Packard MG, McGaugh JL (1996) Inactivation of hippocampus or caudatenucleus with lidocaine differentially affects expression of place and re-sponse learning. Neurobiol Learn Mem 65:65–72.

Paxinos G, Watson C (1986) The rat brain in stereotaxic coordinates, Ed 2.Orlando, FL: Academic.

Perry T, Hodges H, Gray JA (2001) Behavioural, histological and immuno-cytochemical consequences following 192 IgG-saporin immunolesions ofthe basal forebrain cholinergic system. Brain Res Bull 54:29 – 48.

Ragozzino ME, Wenk GL, Gold PE (1994) Glucose attenuates a morphine-induced decrease in hippocampal acetylcholine output: an in vivo micro-dialysis study in rats. Brain Res 655:77– 82.

Ragozzino ME, Unick KE, Gold PE (1996) Hippocampal acetylcholine re-lease during memory testing in rats: augmentation by glucose. Proc NatlAcad Sci USA 93:4693– 4698.

Ragozzino ME, Pal SN, Unick K, Stefani MR, Gold PE (1998) Modulation ofhippocampal acetylcholine release and spontaneous alternation scores byintrahippocampal glucose injections. J Neurosci 18:1595–1601.

Restle E (1957) Discrimination of cues in mazes: a resolution of the “place-vs.-response” question. Psychol Rev 64:217–228.

Stancampiano R, Cocco S, Cugusi C, Sarais L, Fadda F (1999) Serotonin andacetylcholine release response in the rat hippocampus during a spatialmemory task. Neuroscience 89:1135–1143.

Tolman EC, Ritchie BF, Kalish D (1946) Studies in spatial learning. II. Placelearning versus response. J Exp Psychol 36:221–229.

Tolman EC, Ritchie BF, Kalish D (1947) Studies in spatial learning. V. Re-sponse versus place learning by the noncorrection method. J Exp Psychol37:285–292.

Wenk GL (1997) The nucleus basalis magnocellularis cholinergic system:one hundred years of progress. Neurobiol Learn Mem 67:85–95.

Westerink BHC, Timmerman W (1999) Do neurotransmitters sampled bybrain microdialysis reflect functional release? Anal Chim Acta 379:263–274.

White NM, McDonald RJ (2002) Multiple parallel memory systems in thebrain of the rat. Neurobiol Learn Mem 77:125–184.

White NM, Wallet PA (2000) Dorsal hippocampal function in unreinforcedspatial learning. Hippocampus 10:226 –235.

Willingham DB (1998) A neuropsychological theory of motor skill learning.Psychol Rev 105:558 –584.

Woolf NJ (1991) Cholinergic systems in mammalian brain and spinal cord.Prog Neurobiol 37:475–524.

Woolf NJ, Butcher LL (1981) Cholinergic neurons in the caudate-putamencomplex proper are intrinsically organized: a combined Evans blue andacetylcholinesterase analysis. Brain Res Bull 7:487–507.

Wrenn CC, Wiley RG (1998) The behavioral functions of the cholinergic basalforebrain: lessons from 192 IgG-saporin. Int J Dev Neurosci 16:595–602.

Chang and Gold • Switching Memory Systems J. Neurosci., April 1, 2003 • 23(7):3001–3005 • 3005

Nature © Macmillan Publishers Ltd 1998

8

Imagingunconscioussemantic primingStanislas Dehaene*, Lionel Naccache*, Gurvan Le Clec’H*,Etienne Koechlin*, Michael Mueller*,Ghislaine Dehaene-Lambertz†,Pierre-Francois van de Moortele‡ & Denis Le Bihan‡

* INSERM U.334, Service Hospitalier Frederic Joliot, CEA/DRM/DSV,4 Place du General Leclerc, 91401 Orsay, France† Laboratoire de Sciences Cognitives et Psycholinguistique, EHESS/CNRS,75270 Paris cedex 06, France‡ Service Hospitalier Frederic Joliot, CEA/DRM/DSV, 91401 Orsay, France. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Visual words that are masked and presented so briefly that theycannot be seen may nevertheless facilitate the subsequent processingof related words, a phenomenon called masked priming1,2. It hasbeen debated whether masked primes can activate cognitiveprocesses without gaining access to consciousness3–5. Here we use acombination of behavioural and brain-imaging techniques toestimate the depth of processing of masked numerical primes. Ourresults indicate that masked stimuli have a measurable influenceon electrical and haemodynamic measures of brain activity. Whensubjects engage in an overt semantic comparison task with aclearly visible target numeral, measures of covert motor activityindicate that they also unconsciously apply the task instructionsto an unseen masked numeral. A stream of perceptual, semanticand motor processes can therefore occur without awareness.

We presented a numeral between 1 and 9, the prime, to subjectsfor a very short duration (43 ms). The prime was masked by twononsense letter strings, and followed by another numeral, the target(Fig. 1). Under these conditions, even when subjects focused theirattention on the prime, they could neither reliably report itspresence or absence nor discriminate it from a nonsense string(Table 1). Nevertheless, we show here that the prime is processed toa high cognitive level.

We asked subjects to perform a simple semantic categorizationtask on the target numeral. Subjects were asked to press a responsekey with one hand if the target was larger than 5, and with the otherhand if the target was smaller than 5. Unknown to them, each targetnumber was preceded by a masked prime which was varied fromtrial to trial so it too could be larger or smaller than 5. In some trialsthe prime was congruent with the target (both numbers fell on thesame side of 5), and in other trials it was incongruent (one numberbeing larger than 5, and the other being smaller; Fig. 1). We firstestablished that prime–target congruity has a significant influence onbehavioural, electrical and haemodynamic measures of brain function.We then showed that the interference between prime and target canbe attributed to a covert, prime-induced activation of motor cortex,a response bias that must be overcome in incongruent trials. Thisindicates that the prime was unconsciously processed according totask instructions, all the way down to the motor system.

The effect of prime–target congruity on behaviour is shown inFig. 2. Subjects responded more slowly in incongruent trials than incongruent trials (P , 0:0001). All 12 subjects showed a positivepriming effect, ranging from 2 to 43 ms (average 24 ms, s.d.13.5 ms). Furthermore, the entire response time distribution wasshifted by ,24 ms in incongruent trials compared with congruenttrials. Thus, there was no evidence that the effect was found in only asmall proportion of subjects or a small number of trials with adistinct distribution of response times.

Two characteristics of the priming effect link it to a semantic levelof analysis. First, we varied the notations used for the prime numberand for the target number independently. Either number could bepresented as an arabic digit (for example, 1 or 4) or as a written word(for example, ONE or FOUR). Although the target notation had asignificant main effect on response times (P , 0:0001), the amountof priming itself was similar under all conditions of notation anddid not interact with target notation, prime notation, or notationchange. Priming remained significant even when the notations ofthe prime and target numbers differed (for example, prime 4, targetONE; P ¼ 0:0006). Thus, priming occurred at a notation-indepen-dent level of numerical representation6. Second, priming remainedsignificant even after trials with repeated numbers (for example,

letters to nature

NATURE | VOL 395 | 8 OCTOBER 1998 | www.nature.com 597

Figure 1 Experimental design. In each trial, the following four visual stimuli

appeared successively, centred on the same screen location: a random-letter-

string mask, a prime number, another mask, and a target number. Subjects were

not informed of the presence of the prime. They were simply told that a ‘signal’

preceded the target number that they had to classify as larger or smaller than 5.

Half of the trials were of the congruent type (prime and target both falling on the

same side of 5), and half were incongruent.

Table 1 Experimental measures of prime awareness

Prime duration (ms)

0 29 43 57 114 200.............................................................................................................................................................................

Task 1

Hit rate (%) 4.2 10.4 12.5 25.0 85.4 97.9False alarms (%) 4.2 7.3 7.3 3.1 5.2 1.0d’ 0.0 0.20 0.30 1.19** 2.68** 4.36**.............................................................................................................................................................................

Task 2

Hit rate (%) 28.6 40.2 49.1 46.4 78.6 95.5False alarms (%) 34.8 32.1 41.1 30.4 28.6 16.1d’ −0.17 0.22 0.20 0.42* 1.36** 2.69**.............................................................................................................................................................................In two control experiments, subjects were fully informed of the precise structure of thestimuli and were then presented with trials with numerical primes intermixed with trials inwhich the primes were omitted (explicit detection, task 1; six subjects, 96 trials per cell) orreplaced by random strings with the same number of characters (number versus letter-string discrimination, task 2; seven subjects,112 trials per cell). Prime duration was system-atically varied. At the prime duration used in the main experiments (43ms), subjectsconsistently reported not seeing the numerical primes (task 1), did not respond differentlyto prime-absent and prime-present trials (task 1) and were unable to discriminate numericalprimes from letter strings (task 2). Discrimination performance, as measured by d’, a bias-free measure of stimulus discriminability derived from signal-detection theory, began todeviate from chance only for a prime duration of 57ms or more (x2 test; asterisk indicatesP , 0:05; double asterisk indicates P , 0:001).

Nature © Macmillan Publishers Ltd 1998

8

prime ONE, target 1) were excluded from the analysis (P ¼ 0:001).Hence, priming did not simply reflect a word repetition effect, butdepended on the similarity of the prime and target numbers at asemantic level (larger or smaller than 5).

Event-related potentials (ERPs) recorded during the task alsoshowed a prime–target congruity effect. The central positivity,which culminated at about the time of the subject’s response andis thought to index post-perceptual processing7, was delayed by,24 ms in incongruent trials compared with congruent trials(Fig. 3a). We proposed that the slower responses in incongruenttrials might be due to response competition. Subjects wouldunconsciously apply the task instructions to the prime, wouldtherefore categorize it as smaller or larger than 5, and would evenprepare a motor response appropriate to the prime (Fig. 1). Inincongruent trials, this prime-induced covert motor activationwould mismatch with the overt response required to the target,resulting in response competition and hence slower response timesrelative to congruent trials.

According to this theory, we should be able to detect an earlycovert motor activation on the correct response side during con-gruent trials, and on the incorrect response side during incongruenttrials. We tested this prediction using the lateralized readinesspotential (LRP), an ERP measure that indexes the activation oflateralized motor circuits8 (Fig. 3b). The LRP can detect low levels ofcovert response activation that do not necessarily result in an overtmotor response8–10. As it unfolds in time, the LRP departs from zeroas soon as task-relevant information becomes available to bias themotor response towards the left or right hand. Conventionally,positive deflections indicate response preparation on the correctside, whereas negative deflections indicate a transient covert activa-tion on the incorrect response side. Here, the LRP revealed thepredicted covert prime-induced motor preparation. In response-locked averages, before the large positive deviation which reflectedthe activation of motor circuits on the correct response side, theLRP showed a significant difference in the predicted directionbetween congruent and incongruent trials (one-tailed P ¼ 0:015).In incongruent trials, the LRP showed a significant negative devia-tion (one-tailed P ¼ 0:005), indicating motor preparation on theincorrect side of response, whereas in congruent trials a nonsigni-ficant positive deviation occurred (one-tailed P ¼ 0:22). Hence, aperiod of covert prime-induced response competition preceded theovert execution of the correct response.

Covert prime processing could be observed even more directly.

Because the primes and targets were varied independently in theexperimental list, we could test their impact on ERP recordingsseparately. Primes that induced a covert left-hand or right-hand biasproduced a distinct pattern of brain activity over the left and rightmotor cortices (Fig. 4). Primes that were associated with the lefthand during a given block caused a contralateral right-hemisphericnegativity, and primes that were associated with the right handcaused a left-hemispheric negativity. This covert motor primingeffect had a similar scalp topography to the overt motor effect thatwas found before the actual motor response to the target, but it wassmaller and arrived earlier.

Because ERPs have a notoriously imprecise spatial resolution, wewanted to confirm that covert priming originated at least in partfrom motor circuits using the spatially accurate method of func-tional magnetic resonance imaging (fMRI). Response timesrecorded during fMRI recording replicated the prime–target con-gruity effect (P ¼ 0:0027; effect size 20 ms). In fMRI, however, trialswere now separated by 14 s, during which the rise and fall of thehaemodynamic signal was measured in the whole brain every 2 s.

letters to nature

598 NATURE | VOL 395 | 8 OCTOBER 1998 | www.nature.com

Figure 2 Behavioural priming effect. a, Average correct response times recorded

during the ERP experiment are plotted as a function of prime–target congruity for

different prime and target notations (A, Arabic; V, verbal). b, The distribution of

correct responses showed a rightward shift in incongruent trials relative to

congruent trials (bin size 20ms).

Figure 3 ERP measures of prime–target congruity. a, A late positivity (P3)

recorded from the vertex from electrode Cz showed a significant delay in

incongruent trials relative to congruent trials (shaded areas, P , 0:05). b, Deriva-

tion of the lateralized readiness potential (LRP). Individual trials were averaged in

synchrony with the time of the key press, thus suppressing the effects of

response delays. Electrodes C3 and C4 showed large voltage differences in

opposite directions (top two graphs) before left-hand versus right-hand

responses, reflecting the activation of the underlying motor circuits. The LRP

(bottom) is the average of the differences at C3 and at C4, calculated according to

the formula shown. In incongruent trials, before the main positive-going wave-

form reflecting overt response preparation, the LRP was significantly more

negative than in congruent trials (shaded area, P , 0:05), reflecting covert

motor priming.

Nature © Macmillan Publishers Ltd 1998

8

Although this coarse haemodynamic measure cannot resolve thesmall activation delays associated with masked priming, we rea-soned that it should be proportional to the total brain activityaccumulated during a given trial, and should therefore reflect thesum of overt and covert activation in motor areas. We thereforeextracted the fMRI signal profile from the left and right motorcortices, and used it to derive an index of lateralized motoractivation analogous to the LRP, the lateralized bold response(LBR; Fig. 5). This measure showed a highly significant positivepeak following each motor response. The LBR was smaller inincongruent trials than in congruent trials. The direction of thiseffect is identical to that seen in the LRP (Fig. 3b). Both effectsindicate a significant prime-induced activation on the wrongresponse side on incongruent trials relative to congruent trials,diminishing the overall size of the activation on the correct motorside. Electrical and haemodynamic measures of covert maskedpriming were complementary: fMRI localized the priming effectto motor cortex, but was insensitive to its time course, whereas ERPspinpointed the priming effect to a small window of time before theovert target-related motor activation.

Our results resolve the issue of the depth of processing of maskedprimes3. First, the results show that the processing of masked primes isaccompanied by measurable modifications of electrical brain activityand of cerebral blood flow. This concurs with the observation of amodulation of amygdala activity by masked visual faces11,12. As shownpreviously13,14, brain imaging now has the potential to image uncon-scious cerebral processing. Second, unconscious activity is not con-fined to brain areas involved in sensory processing. Even areas involvedin motor programming were covertly activated here, depending on theside of the motor response that subjects should have made if they hadresponded to the primes according to the task instructions. Becausethis motor parameter was determined by whether the prime was

larger or smaller than 5, the prime must have been categorized at thesemantic level. By showing that a large amount of cerebral processing,including perception, semantic categorization and task execution,can be performed in the absence of consciousness, our resultsnarrow down the search for its cerebral substrates. M. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Methods

Procedure. All experiments were approved by the French ethical committeefor biomedical research, and subjects gave informed consent. The stimulus setconsisted of 64 pairs of prime and target numbers 1, 4, 6 and 9, each in eitherArabic or spelled-out format. Subjects performed the number-comparison tasktwice in counterbalanced order. In one block, the instruction was to press theright-hand key for targets larger than 5 and the left-hand key for targets smallerthan 5. In another block, the opposite instruction was used. Within each block,subjects received initial training (ERPs, 16 trials; fMRI, 25 trials) before theexperimental session (ERPs, 256 trials; fMRI, 64 trials).Event-related potentials. Twelve subjects were tested (six males; mean age 25;one other subject was rejected because of excessive motion). We presented atotal of 512 stimuli at a 3-s rate on a standard PC-compatible SVGA screen(EGA mode, 70 Hz refresh rate). The electroencephalogram was digitized at125 Hz from 128 scalp electrodes referenced to the vertex15, for a 2,048-msperiod starting 400 ms before the onset of the first mask. We rejected trials withincorrect responses, voltages exceeding 6100 mV, transients exceeding 650 mV,electro-oculogram activity exceeding 670 mV, or response times outside a 250–1,000-ms interval. The remaining trials were averaged in synchrony either withstimulus or with response onset, digitally transformed to an average reference,band-pass filtered (0.5–20 Hz), and corrected for baseline over a 400-mswindow before stimulus onset (similar results were observed with the raw,unfiltered data). Experimental conditions were compared by sample-by-sample t-tests on electrodes C3, C4 and Cz, with a criterion of P , 0:05 forfive consecutive samples. Two-dimensional maps of scalp voltage and t-valueswere constructed by spherical spline interpolation16.

letters to nature

NATURE | VOL 395 | 8 OCTOBER 1998 | www.nature.com 599

Figure 4 ERP measures of covert and overt motor activation. At each electrode

site, two independent t-tests were performed on scalp voltages. The first test

compared trials with overt target-induced right-hand or left-hand responses

(green curve and top right map). The second test compared trials with primes

inducing a covert left-hand or right-hand bias (orange curve and top left map).

Statistical parameter maps in polar coordinates (top) show colour-coded t-test

values at each site on the scalp, at the delay at which the effect was maximal. The

sign and topography of the covert motor priming effect and of the overt motor

responseeffect are similar, indicating that primesand targets wereprocessed in a

similar way according to task instructions. The bottom curve shows the temporal

evolution of the two independent t-tests at electrode site C3, positioned over the

left motor cortex (coloured areas, P , 0:05). The covert effect preceded the overt

effect by 152ms, a delay roughly comparable to the interval between the onsets of

the prime and target (114ms).

Nature © Macmillan Publishers Ltd 1998

8

Functional magnetic resonance imaging. Nine new subjects were tested(seven males; mean age 26). We used an event-related design17. We presented alist of 128 randomly intermixed stimuli through mirror glasses and an activematrix video projector (EGA mode, 70 Hz refresh rate), with a 14-s inter-stimulus interval. In each trial, stimulus onset was synchronized with the

acquisition of the first slice in a series of seven volumes of eighteen slices each.We used a gradient-echo echo-planar imaging sequence sensitive to brainoxygen-level-dependent contrast (18 contiguous axial slices, 6-mm thickness,repetition time=echo time ¼ 2;000=40 ms, in-plane resolution 3 3 4 mm2,64 3 64 matrix) on a 3-Tesla whole-body system (Bruker). High-resolutionanatomical images (three-dimensional gradient-echo inversion-recoverysequence, inversion time ¼ 700 ms, repetition time ¼ 1;600 ms, field ofview ¼ 192 3 256 mm2, matrix ¼ 256 3 128 3 256, slice thickness ¼ 1 mm)were also acquired.

Analysis was done with SPM96 software. Images were corrected for subjectmotion, normalized to Talairach coordinates using a linear transformcalculated on the anatomical images, smoothed (full-width athalf-maximum ¼ 15 mm), and averaged to define four types of event (con-gruent or incongruent × left-hand or right-hand response). Images from allnine subjects were then analysed together. We used the generalized linear modelto model the intensity level of each pixel as a linear combination, for eachsubject and each event type, of two activation functions with haemodynamiclags 4 and 7 s, thus allowing for differences in acquisition and activation timesacross slices and brain regions. We used a voxelwise significance level of 0.001,corrected to P , 0:05 for multiple comparisons across the brain volume.

Received 21 May; accepted 17 July 1998.

1. Marcel, A. J. Conscious and unconscious perception: experiments on visual masking and wordrecognition. Cogn. Psychol. 15, 197–237 (1983).

2. Forster, K. I. & Davis, C. Repetition priming and frequency attenuation in lexical access. J. Exp.Psychol. Learn. Mem. Cogn. 10, 680–698 (1984).

3. Holender, D. Semantic activation without conscious identification in dichotic listening, parafovealvision and visual masking: a survey and appraisal. Behav. Brain Sci. 9, 1–23 (1986).

4. Cheesman, J. & Merikle, P. M. Priming with and without awareness. Percept. Psychophys. 36, 387–395(1984).

5. Merikle, P. M. Perception without awareness: critical issues. Am. Psychol. 47, 792–796 (1992).6. Dehaene, S. & Akhavein, R. Attention, automaticity and levels of representation in number

processing. J. Exp. Psychol. Learn. Mem. Cogn. 21, 314–326 (1995).7. McCarthy, G. & Donchin, E. A metric for thought: a comparison of P300 latency and reaction time.

Science 211, 77–80 (1981).8. Coles, M. G. H., Gratton, G. & Donchin, E. Detecting early communication: using measures of

movement-related potentials to illuminate human processing. Biol. Psychol. 26, 69–89 (1988).9. Miller, J. O. & Hackley, S. A. Electrophysiological evidence for temporal overlap among contingent

mental processes. J. Exp. Psychol. Gen. 121, 195–209 (1992).10. van Turennout, M., Hagoort, P. & Brown, C. M. Brain activity during speaking: from syntax to

phonology in 40 milliseconds. Science 280, 572–574 (1998).11. Whalen, P. J. et al. Masked presentations of emotional facial expressions modulate amygdala activity

without explicit knowledge. J. Neurosci. 18, 411–418 (1998).12. Morris, J. S., Ohman, A. & Dolan, R. J. Conscious and unconscious emotional learning in the human

amygdala. Nature 393, 467–470 (1998).13. Berns, G. S., Cohen, J. D. & Mintun, M. A. Brain regions responsive to novelty in the absence of

awareness. Science 276, 1272–1275 (1997).14. Sahraie, A. et al. Pattern of neuronal activity associated with conscious and unconscious processing of

visual signals. Proc. Natl Acad. Sci. USA 94, 9406–9411 (1997).15. Tucker, D. Spatial sampling of head electrical fields: the geodesic electrode net. Electroencephalogr.

Clin. Neurophysiol. 87, 154–163 (1993).16. Perrin, F., Pernier, J., Bertrand, D. & Echallier, J. F. Spherical splines for scalp potential and current

density mapping. Electroencephalogr. Clin. Neurophysiol. 72, 184–187 (1989).17. Buckner, R. L. et al. Detection of cortical activation during averaged single trials of a cognitive task

using functional magnetic resonance imaging. Proc. Natl Acad. Sci. USA 93, 14878–14883 (1996).

Correspondence and requests for materials should be addressed to S.D. (e-mail: dehaene@shfj.cea.fr).

letters to nature

600 NATURE | VOL 395 | 8 OCTOBER 1998 | www.nature.com

Figure 5 fMRI measure of motor priming. Top, voxels that showed significant

differences in bold signal intensity between overt left-hand and right-hand

responses are coded using the colour scale at left. The two voxels with the most

significant overt motor effects were located in the left and right precentral cortex

(Talairach coordinates −39, −21, 66 and 39, −15, 63). Centre, plots of the average

fMRI signal of the two voxels as a function of time show activation for contralateral

movements followinga haemodynamic delayof about 4–8 s (planned contrast for

an increase in left–right differences from baseline (time points 0 and 2s) to

activated state (time points 4, 6 and 8 s): left motor cortex, Fð1; 8Þ ¼ 20:1,

P ¼ 0:0021; right motor cortex, Fð1; 8Þ ¼ 27:0, P ¼ 0:0001). Bottom, it was therefore

possible to construct an fMRI measure similar to the event-related LRP, indexing

the time course of overt motor activation, which we term the lateralized bold

response (LBR). The LBR was significantly larger (+9%) in congruent trials than in

incongruent trials (interaction of the congruity factor with the baseline-to-

activated-state contrast, Fð1; 8Þ ¼ 6:23, P ¼ 0:037).

Local control of informationflow insegmental andascendingcollateralsof single afferentsJ. Lomelı, J. Quevedo, P. Linares & P. Rudomin

Department of Physiology, Biophysics and Neurosciences,Centro de Investigacion y de Estudios Avanzados del Instituto PolitecnicoNacional, Apartado Postal 14-740, Mexico D.F. 07000, Mexico. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

In the vertebrate spinal cord, the activation of GABA(g-amino-butyric acid)-releasing interneurons that synapse with intraspinalterminals of sensory fibres leading into the central nervous system(afferent fibres) produces primary afferent depolarization andpresynaptic inhibition1–3. It is not known to what extent these

Integration of Visual and Linguistic Information in Spoken Language ComprehensionAuthor(s): Michael K. Tanenhaus, Michael J. Spivey-Knowlton, Kathleen M. Eberhard and JulieC. SedivySource: Science, New Series, Vol. 268, No. 5217 (Jun. 16, 1995), pp. 1632-1634Published by: American Association for the Advancement of ScienceStable URL: http://www.jstor.org/stable/2888637 .

Accessed: 02/11/2013 05:40

Your use of the JSTOR archive indicates your acceptance of the Terms & Conditions of Use, available at .http://www.jstor.org/page/info/about/policies/terms.jsp

.JSTOR is a not-for-profit service that helps scholars, researchers, and students discover, use, and build upon a wide range ofcontent in a trusted digital archive. We use information technology and tools to increase productivity and facilitate new formsof scholarship. For more information about JSTOR, please contact support@jstor.org.

.

American Association for the Advancement of Science is collaborating with JSTOR to digitize, preserve andextend access to Science.

http://www.jstor.org

This content downloaded from 130.226.229.16 on Sat, 2 Nov 2013 05:40:24 AMAll use subject to JSTOR Terms and Conditions

sisted of synthetic peptide conjugated to bovine thy- roglobulin with the use of glutaraldehyde. The antiser- um recognized a protein band of 80 to 90 kD on protein immunoblots of membranes prepared from cells transfected with the rat SPR [S. R. Vigna et a!., J. Neurosci. 14, 834 (1994)]. The cells could be immu- nostained with the antiserum, and the staining could be blocked by preabsorbing the antiserum with SPR393_407. After the incubation with the primary an- tibody, the tissue sections were washed for 30 min at 22?C in PBS (pH 7.4) and then incubated in the sec- ondary antibody solution (pH 7.4) for 2 hours at 22?C. This secondary antibody solution was identical to the primary antibody solution, except that cyanine (Cy3)- conjugated donkey anti-rabbit immunoglobulin G (no. 711-165-152; Jackson ImmunoResearch Labs, West Grove, PA) was present (at a concentration of 1: 600) instead of the SPR antibody. Finally, the tissue sec- tions were washed for 20 min in PBS (pH 7.4, 22?C), mounted on gelatin-coated slides, and coverslipped with PBS-glycerine containing 1.0% p-phenylenedia- mine to reduce photobleaching.

12. To examine the sites of internalization within the cell, sections were examined with an MRC-600 Confocal Imaging System (Bio-Rad, Boston, MA) and with an Olympus BH-2 microscope equipped for epifluores- cence (Olympus America, Lake Success, NY). Both of the microscopes were set up as previously de- scribed [T. C. Brelje, D. W. Scharp, R. L. Sorenson, Diabetes 38, 808 (1989); P. W. Mantyh et a!., J. Neurosci. 15, 152 (1995)]. To determine the percent- age of neurons showing intense internalization, sec- tions were examined with a Leitz Orthoplan II micro- scope (Ernst Leitz Wetzler, Germany) equipped for fluorescence. To determine the total number of cell bodies showing SPR-positive internalization, immu- nostained sections were viewed through a 1.0-cm2 eyepiece grid that was divided into 100 1-mm by 1-mm units, and the total number of SPR-positive cell bodies and the number of SPR-positive cell bod- ies showing significant SPR-positive internalization were counted. SP-induced internalization has been shown to be blocked by nonpeptide SPR antago- nists in transfected cells [A. M. Garland, E. F. Grady, D. G. Payan, S. R. Vigna, N. W. Bunnett, Biochem. J. 303, 1 77 (1994)] and in cultured spinal neurons (19).

13. To determine whether the capsaicin-induced SPR internalization in lamina I neurons was dose depen- dent, 0.1, 1.0, 100, and 300 pg of capsaicin was injected into one hindpaw of rats. The percentage of neurons showing significant internalization (>20 en- dosomes per cell body) was examined 1 min later. The percentage of L4 lamina I neurons showing sig- nificant internalization was 26, 54, 55, and 70% for 0.1, 1 .0, 100, and 300 pg of capsaicin, respectively.

14. We have consistently observed an increase in total apparent SPR immunoreactivity per cell or per field after either exogenous injection of SP in the spinal cord (19) or after increased release of endogenous SP as observed here. Although this observation may re- flect a change in the accessibility of the antiserum to the SPR or a change in SPR immunoreactivity related to possible changes in receptor conformation after agonist binding, both of these possibilities seem un- likely. The most plausible explanation is that the in- crease in total SPR-positive immunoreactivity ob- served after SP binding simply reflects the concentra- tion of SPRs into small membrane-bound endo- somes, which are present in confocal slices in greater amounts than is plasma membrane, in which the SPR is located before peptide binding. This interpretation is supported by the observation that an early step in agonist-induced endocytosis is the migration of re- ceptors in the plane of the plasma membrane to pits (2), thereby effectively concentrating the receptors in the membranes destined to become endosomal.

15. J. Jack, in The Neurosciences: Fourth Study Pro- gram, F. 0. Schmitt and F. G. Worden, Eds. (MIT Press, Cambridge, MA, 1979), pp. 423-437; B. Katz, Nerve, Muscle and Synapse (McGraw-Hill, New York, 1966), pp. 12-51.

16. M. N. Castel, J. Woulfe, X. Wang, P. M. Laduron, A. Beaudet, Neuroscience 50, 269 (1992).

17. C. J. Woolf, Nature 306, 686 (1983); P. Dubner, in Proceedings of VI World Congress on Rain, Rain Research and Clinical Management, M. Bond, E.

Chariton and C. J. Woolf, Eds. (Elsevier, Amsterdam, 1991), vol. 5, pp. 263-270; D. A. Simone et al., J. Neurophysiol. 66, 228 (1991).

18. T. Mitchison and M. Kirschner, Neuron 1, 761 (1988); G. J. Brewer and C. W. Cotman, Neurosci. Lett. 99, 268 (1989); D. Bigot and S. P. Hunt, ibid. 111, 275 (1990); ibid. 131, 21 (1991); D. Bigot, A. Matus, S. P. Hunt, Eur. J. Neurosci. 3, 551 (1991).

19. P. W. Mantyh, unpublished observations.

20. We thank A. Georgopoulos for helpful comments, R. Elde and the Biomedical Imaging Processing Labo- ratory for use of the confocal microscope, and C. Garret for a gift of RP-67580. Supported by NIH grants NS23970, NS14627, NS21445, DA08973, NS31223, and GM1 5904 and by a Veterans Admin- istration Merit Review.

30 September 1994; accepted 2 March 1995

Integration of Visual and Linguistic Information in Spoken Language Comprehension

Michael K. Tanenhaus,* Michael J. Spivey-Knowlton, Kathleen M. Eberhard, Julie C. Sedivy

Psycholinguists have commonly assumed that as a spoken linguistic message unfolds over time, it is initially structured by a syntactic processing module that is encapsulated from information provided by other perceptual and cognitive systems. To test the effects of relevant visual context on the rapid mental processes that accompany spoken language comprehension, eye movements were recorded with a head-mounted eye-tracking sys- tem while subjects followed instructions to manipulate real objects. Visual context influ- enced spoken word recognition and mediated syntactic processing, even during the earliest moments of language processing.

The two essential properties of language are that it refers to things in the world and that its grammatical structure can be char- acterized independently of meaning or ref- erence (1). The autonomy of grammatical structure has led to a long tradition in psy- cholinguistics according to which it is as- sumed that the brain mechanisms responsi- ble for the rapid syntactic structuring of continuous linguistic input are "encapsulat- ed" from other cognitive and perceptual systems (2), much as early visual processing often is claimed to be structured by auton- omous processing modules (3). This con- trasts with a second tradition by which language processing is inextricably tied to reference and relevant behavioral context (4). The primary empirical evidence that syntactic processing is modular is that brief syntactic ambiguities, which arise because language unfolds over time, appear to be initially resolved independently of prior context. Unfortunately, it has been impos- sible to perform the crucial test to deter- mine whether strongly constraining nonlin- guistic information can influence the earli- est moments of syntactic processing, be- cause experimental techniques that provide fine-grained temporal information about spoken language comprehension could not be used in natural contexts. However, by recording eye movements (5) as partici- pants followed instructions to move objects

M. K. Tanenhaus, M. J. Spivey-Knowlton, K. M. Eber- hard, Department of Brain and Cognitive Sciences, Uni- versity of Rochester, Rochester, NY 14627, USA. J. C. Sedivy, Department of Linguistics, University of Rochester, Rochester, NY 14627, USA.

*To whom correspondence should be addressed.

(for example, "Put the apple that's on the towel in the box"), we were able to monitor the ongoing comprehension process on a millisecond time scale. This enabled us to observe the rapid mental processes that ac- company spoken language comprehension in natural behavioral contexts in which the language had clear real-world referents.

Our initial experiments demonstrated that individuals processed the instructions incrementally, making saccadic eye move- ments to objects immediately after hearing relevant words in the instruction. Thus the eye movements provided insight into the mental processes that accompany language comprehension. For example, when asked to touch one of four blocks that differed in marking, color, or shape, with instructions such as "Touch the starred yellow square," a person made an eye movement to the target block an average of 250 ms after the end of the word that uniquely specified the target with respect to the visual alternatives (for example, after "starred" if only one of the blocks was starred, and after "square" if there were two starred yellow blocks). With more complex instructions, individuals made in- formative sequences of eye movements that were closely time-locked to words in the instruction that were relevant to establishing reference. In one experiment, subjects were given a complex instruction such as "Put the five of hearts that is below the eight of clubs above the three of diamonds,"' with a display composed of seven miniature playing cards, including two fives of hearts. As the person heard "the five of hearts," she looked at each of the two potential referents successively. After hearing "below the," she immediately

1632 SCIENCE * VOL. 268 * 16 JUNE 1995

This content downloaded from 130.226.229.16 on Sat, 2 Nov 2013 05:40:24 AMAll use subject to JSTOR Terms and Conditions

REPORTS

looked at a ten of clubs, which was above the five of hearts on which she had been fixating. By the time she heard the end of the word "clubs," her eyes moved to in- terrogate the card above the other five of hearts, which was the eight of clubs; thus she identified that five as the target. The eye immediately shifted to the target card and remained there until the hand reached for it.

We also found that the visual context affected the resolution of temporary ambigu- ities within individual words. For example, halfway through the spoken word "candy," the auditory input is consistent with both "candy" and "candle." Subjects were pre- sented with a display of everyday objects that sometimes included two objects with initial- ly similar names (for example, candy and candle) and were instructed to move them around ("Pick up the candy. Now put it above the fork"). The mean time to initiate an eye movement to the correct object (can- dy) was 145 ms from the end of the word when there was not another object with a similar name, compared with 230 ms when an object with a similar name was included in the display (6). Because it takes about 200 ms to program a saccadic eye movement (7), these results demonstrate that the individual identified the object before hearing the end of the word, when none of the other objects had a similar name.

The compelling evidence for the rapid and nearly seamless integration of visual and linguistic information that emerged from these experiments led us to test whether

B

C

A,A

D,B'

"Put the apple on the towel in the box." A B C D

i I i I i r i T rr 0 500 1000 1500 2000 2500

"Put the apple that's on the towel in the box."

0 500 1000 1500 2000 2500 Time (ms)

Fig. 1. Typical sequence of eye movements in the one-referent context for the ambiguous and un- ambiguous instructions. Letters show when in the instruction each eye movement occurred, as de- termined by the mean latency for that type of eye movement (A' and B' correspond to the unambig- uous instruction).

information provided by the visual context would affect the syntactic processing of an instruction. The strongest evidence for the modularity of syntactic processing has come from studies of sentences with brief syntactic ambiguities, in which readers have clear preferences for particular interpretations that persist momentarily even when preceding linguistic context supports the alternative interpretation (8). However, under these conditions, the context may not be immedi- ately accessible, because it has to be repre- sented in memory. We reasoned that a rele- vant visual context that was available for the listener to interrogate as the linguistic input unfolded might influence its initial syntactic analysis. If so, this would provide definitive evidence against syntactic modularity.

We used instructions containing the tem- porary syntactic ambiguity with perhaps the strongest syntactic preference in English, as illustrated by the examples, "Put the apple on the towel in the box," and "Put the apple that's on the towel in the box."

In the first sentence, the first preposition- al phrase, "on the towel," is ambiguous as to whether it modifies the noun phrase ("the apple"), thus specifying the location of the object to be picked up, or whether it denotes the destination, that is, the location where the apple is to be put. Those who experiment with this type of ambiguity consistently find that readers and listeners interpret the first prepositional phrase as specifying the desti- nation, which results in momentary confu- sion when they encounter the second prep- osition ("in") (9). Modular models attribute this preference to syntactic simplicity and to

B

"Put the apple on the towel in the box." A B C

o 500 1000 1500 2000 2500

"Put the apple that's on the towel in the box." A B C i I i i i i

0 500 1000 1500 2000 2500 Time (ms)

Fig. 2. Typical sequence of eye movements in the two-referent context. Note that the sequence and the timing of eye movements, relative to the nouns in the speech stream, did not differ for the ambig- uous and unambiguous instructions.

the syntactic requirements of the verb "put" (10). In the second sentence, the word "that's" disambiguates the phrase as a mod- ifier and serves as an unambiguous control condition.

Six people who had not performed these tasks before were presented with three in- stances of each of the four conditions created by pairing the two types of instructions (am- biguous and unambiguous), as illustrated in the example above, with a one-referent vi- sual context that supported the destination interpretation and a two-referent context that supported the modification interpreta- tion (11). In the one-referent context for that example, the workspace contained a towel with an apple on it, another towel without an apple, a box, and a pencil. Upon hearing the phrase "the apple," individuals can immediately identify the object to be moved because there is only one apple, and thus they are likely to assume that "on the towel" is specifying the destination. In the two-referent context, the pencil was re- placed by a second apple that was on a napkin. Thus "the apple" could refer to ei- ther of the two apples, and the phrase "on the towel" provides modifying information that specifies which apple is the correct ref- erent (12). However, if initial syntactic pro- cessing is encapsulated, as modular theories claim, then people should still initially inter- pret "on the towel" as the destination.

In fact, strikingly different fixation pat- tems between the two visual contexts re- vealed that the ambiguous phrase "on the towel" was initially interpreted as a destina- tion in the one-referent context, but as a modifier in the two-referent context. In the one-referent context with the ambiguous in- struction, participants first looked at the tar- get object (the apple) 500 ms after hearing "apple," but then they looked at the incor- rect destination (the irrelevant towel) 55% of the time, shortly after hearing "towel"; this indicated that they had initially inter- preted "on the towel" as specifying the des- tination. The participants then looked back

C 0

1.0

Instruction

0.8 B Ambiguous U Unambiguous

a 0.6-

0.

.0)

oo6

2 0.2-

e0 th

I'

One-referent Two-referent context context

Fg Proportion of trials in which participants loeatteincorrect destination.

SCIENCE * VOL. 268 * 16 JUNE 1995 1633

This content downloaded from 130.226.229.16 on Sat, 2 Nov 2013 05:40:24 AMAll use subject to JSTOR Terms and Conditions

at the apple to pick it up, and finally at the box for placement. When the unambiguous instruction was presented in the one-referent context, participants never looked at the incorrect destination (13) (Fig. 1).

In the two-referent context, participants often looked at both apples shortly after hearing "the apple," which reflected the fact that reference could not be established on the basis of just that input. Participants looked at the incorrect referent during 42% of the unambiguous trials and during 61 % of the ambiguous trials. [In contrast, in the one-referent context, in which reference could be established given just "the apple," individuals rarely looked at the incorrect object (pencil); this occurred during 0 and 6% of the trials for the ambiguous and un- ambiguous instructions, respectively.] The time it took participants to establish refer- ence correctly in the two-referent context did not differ for the ambiguous and unam- biguous instructions, which indicates that "on the towel" was immediately interpreted as a modifier, not as a destination. Individ- uals then typically looked directly to the box for object placement without looking at the incorrect destination (Fig. 2). In contrast with the one-referent context, ambiguity in the instruction did not affect the proportion of eye movements to the incorrect destination in the two-referent context (14) (Fig. 3).

Our results demonstrate that in natural contexts, people seek to establish reference with respect to their behavioral goals during the earliest moments of linguistic processing. Moreover, referentially relevant nonlinguis- tic information immediately affects the man- ner in which the linguistic input is initially structured. Given these results, approaches to language comprehension that assign a central role to encapsulated linguistic sub- systems are unlikely to prove fruitful. More promising are theories by which grammatical constraints are integrated into processing systems that coordinate linguistic and non- linguistic information as the linguistic input is processed (10, 15). Finally, our results show that with well-defined tasks, eye move- ments can be used to observe under natural conditions the rapid mental processes that underlie spoken language comprehension. This paradigm can be extended to explore questions on topics ranging from recognition of spoken words to conversational interac- tions during cooperative problem solving.

REFERENCES AND NOTES

1. N. Chomsky, Aspects of the Theory of Syntax (MIT Press, Cambridge, MA, 1965); S. Pinker, Science 253, 530 (1991); The Language Instinct (Morrow, New York, 1994).

2. J. A. Fodor, Modularity of Mind (MIT Press, Cam- bridge, MA, 1983).

3. Early stages of visual information processing appear to segregate different features of visual input, such as

form, color, motion, and depth, both anatomically and functionally, presumably to increase speed and efficiency in early computation [M. Livingstone and D. Hubel, Science 240, 740 (1988)].

4. H. Clark, Arenas of Language Use (Univ. of Chicago Press, Chicago, 1994); W. D. Marslen-Wilson, Na- ture 244, 522 (1973); Science 189, 226 (1975).

5. We monitored eye movements with an Applied Sci- entific Laboratories camera that was mounted on a lightweight helmet. The camera provides an infrared image of the eye at 60 Hz. The center of the pupil and the corneal reflection are tracked to determine the orbit of the eye relative to the head. Accuracy is better than 1 degree of arc, with virtually unrestricted head and body movements. For details, see D. Bal- lard, M. Hayhoe, J. Pelz, J. Cog. Neurosci. 7, 66 (1995). Instructions were spoken into a microphone connected to a Hi-8 VCR that also recorded the field of view and eye position of the participant.

6. Eight objects were on a table with a center fixation cross. Each trial began with the instruction, "Look at the cross." The eye-movement latency difference be- tween the conditions with and without objects with similar names was reliable [t(7) = 3.04, P < 0.02].

7. E. Matin, K. Shao, K. Boff, Percept. Psychophys. 53, 372 (1993).

8. For review, see L. Frazier, in Attention & Performance XII, M. Coltheart, Ed. (Lawrence Erlbaum, Hove, UK, 1987), pp. 559-586.

9. F. Ferreira and C. Clifton, J. Mem. Lang. 25, 348 (1 986); M. Britt, ibid. 33, 251 (1994).

10. For review, see M. Spivey-Knowlton and J. Sedivy, Cognition, in press.

11. The 12 critical instructions were embedded among 90 filler instructions. Each trial began with the com- mand, "Look at the cross."

12. S. Crain and M. Steedman [in Natural Language Parsing, D. Dowty, L. Kartunnen, H. Zwicky, Eds. (Cambridge Univ. Press, Cambridge, 1985), pp.

320-358] and G. Altmann and M. Steedman [Cog- nition 30, 191 (1988)] have developed a theory of syntactic ambiguity resolution in which referential context is central.

13. This difference between ambiguous and unambigu- ous instructions was reliable by a planned compari- son [t(5) = 4.11, P < 0.01].

14. The interaction between context and ambiguity for eye movements to the incorrect destination was reli- able [F(1,5) = 8.24, P < 0.05]. Also, a three-way interaction between context, ambiguity, and type of incorrect eye movement (to object or to destination) revealed the bias toward a destination interpretation in the one-referent context and toward a modification interpretation in the two-referent context [F(1,5) 18.41, P < 0.01].

15. J. McClelland, in Attention and Performance XII, M. Coltheart, Ed. (Lawrence Erlbaum, Hove, UK, 1987), pp. 3-36; R. Jackendoff, Languages of the Mind (Bradford, Cambridge, MA, 1992); C. Pollard and 1. Sag, Head-Driven Phrase Structure Grammar (Univ. of Chicago Press, Chicago, 1993); M. MacDonald, N. Pearlmutter, M. Seidenberg, Psychol. Rev. 101, 676 (1994); M. Tanenhaus and J. Trueswell, in Handbook of Cognition and Perception, J. Miller and P. Eimas, Eds. (Academic Press, San Diego, CA, in press).

16. We thank D. Ballard and M. Hayhoe for encouraging us to use their laboratory (National Resource Labora- tory for the Study of Brain and Behavior) and for ad- vice on the manuscript, P. Lennie and R. Jacobs for helpful comments, J. Pelz for teaching us how to use the equipment, and K. Kobashi for assistance in data collection. Supported by NIH resource grant 1 -P41 - RR09283, NIH HD27206 (M.K.T.), an NSF graduate fellowship (M.J.S.-K.), and a Social Sciences and Hu- manities Research Council of Canada fellowship (J.C.S.). All participants gave informed consent.

9 January 1995; accepted 4 April 1995

Origins of Fullerenes in Rocks

Naturally occurring fullerenes have been found in rock samples that were subject to singular geologic events such as lightning strokes (1), wildfires at the K-T boundary (2), and meteoritic impacts (3). These find- ings are expected, as fullerenes form normal- ly under highly energetic conditions. How- ever, P. R. Buseck et al. (4) reported the presence of C60 in a carbon-rich rock sample from Shunga, in Karelia, Russia, in which the host geologic unit was highly metamor- phosed and there was no evidence of expo- sure to extreme conditions. If fullerenes did form naturally in such an environment, we would expect them to be widely present else- where, and there would be many ramifica- tions. For example, the presence of fullerenes in the earliest times would have implications for the evolution of life (that is, as an early source of large molecules).

We studied the occurrence and distribu- tion of fuillerenes in carbon-rich rocks, in- cluding samples of shungite from the depos- it in Shunga. To avoid sources of contami- nation by fullerenes, our samples were pre- pared in laboratories where there had been no previous work done on fullerenes. The outer 2- to 4-mm portion of the shungite

samples was removed, and only the core material was gently crushed and ground be- fore mass spectrometry (MS) analysis was carried out directly on the rock powder. Laser Fourier-transform MS and thermal desorption negative ion MS methods were used. In the thermal desorption MS, the temperature was scanned up to 450?C, at which C60 and C70 are fully volatilized. One sample was purposely contaminated with 100 ppm of commercial fullerenes as a con- trol and to check the sensitivity of the analysis. The result of this reference test indicated that we could detect fullerenes at 10 ppm, or less, without difficulty.

The three samples from the Shunga locality (5) had a variable carbon content of about 100, 90, and 10% by weight. These samples were hosted by about 2-bil- lion-year-old metamorphosed volcanic and sedimentary rocks of the Karelian ter- rain, which extends northwest through Finland and into Finnmark (northern Norway). We also analyzed one carbon- rich sample from the Bidjovagge mine near Kautokeino, Finnmark, from rocks with broadly similar age, provenance, and metamorphic history as those of Shunga.

1634 SCIENCE * VOL. 268 * 16 JUNE 1995

This content downloaded from 130.226.229.16 on Sat, 2 Nov 2013 05:40:24 AMAll use subject to JSTOR Terms and Conditions

Psychological Science23(8) 842 –847© The Author(s) 2012Reprints and permission: sagepub.com/journalsPermissions.navDOI: 10.1177/0956797612439070http://pss.sagepub.com

One feature that separates humans from other primates is the propensity to make inferences regarding other individuals’ mental states, and particularly inferences relating to beliefs (Call & Tomasello, 2008; Premack & Woodruff, 1978). For example, imagine that you see a man you do not know walk by and glance at you—twice. You would likely question his inten-tions for doing so and consider that maybe he mistakenly thought he knew you. This ability that allows a person to rea-son about another’s beliefs, feelings, desires, intentions, or goals is termed theory of mind (ToM; Premack & Woodruff, 1978).

Given the important role that such social processing plays in people’s everyday lives, and the benefits it affords individu-als in interacting with others and with the environment, it is not surprising that ToM has been a major topic of investigation in cognitive science. Moreover, the severe limitations encoun-tered by individuals who are impaired in ToM operations (e.g., those with an autism spectrum disorder, or ASD; Baron-Cohen, 1995; Baron-Cohen, Leslie, & Frith, 1985; Frith, 2003) make the understanding of how belief inference oper-ates all the more crucial. A key paradigm for assessing ToM abilities is the Sally-Anne false-belief task (Wimmer & Perner, 1983): In still images, movies, or “live” performance (with puppets, actors, or both), “Sally” sees an object (e.g., a ball) being placed in a container. Sally then leaves the room. Next,

“Anne” hides the object in a different container. When Sally returns to the room, participants are required to identify the location where they think Sally will first look for the object. To succeed at the task, participants must select (e.g., point to) the location that is consistent with Sally’s belief, as opposed to the actual, known location of the object.

Passing this explicit Sally-Anne task is thought to reflect a developmental milestone, which is typically achieved by the age of 4 years (Perner & Lang, 1999). Such findings suggest that children understand other people’s beliefs by this age. How-ever, recent research using a variety of implicit ToM tasks sug-gests that children as young as 7 months may be able to register other individuals’ beliefs (Clements & Perner, 1994; Kovács, Téglás, & Endress, 2010; Onishi & Baillargeon, 2005). For example, monitoring of eye movement behavior in free-viewing false-belief scenarios has demonstrated that 2-year-olds prefer-entially look toward the location at which the actor believes the

Corresponding Authors:Paul E. Dux, School of Psychology, University of Queensland, McElwain Building, St Lucia, Queensland 4072, Australia E-mail: paul.e.dux@gmail.com

Dana Schneider, School of Psychology, University of Queensland, McElwain Building, St Lucia, Queensland 4072, Australia E-mail: msdanaschneider@gmail.com

Cognitive Load Disrupts Implicit Theory-of-Mind Processing

Dana Schneider1, Rebecca Lam1, Andrew P. Bayliss2, and Paul E. Dux1

1University of Queensland and 2University of East Anglia

Abstract

Eye movements in Sally-Anne false-belief tasks appear to reflect the ability to implicitly monitor the mental states of other individuals (theory of mind, or ToM). It has recently been proposed that an early-developing, efficient, and automatically operating ToM system subserves this ability. Surprisingly absent from the literature, however, is an empirical test of the influence of domain-general executive processing resources on this implicit ToM system. In the study reported here, a dual-task method was employed to investigate the impact of executive load on eye movements in an implicit Sally-Anne false-belief task. Under no-load conditions, adult participants displayed eye movement behavior consistent with implicit belief processing, whereas evidence for belief processing was absent for participants under cognitive load. These findings indicate that the cognitive system responsible for implicitly tracking beliefs draws at least minimally on executive processing resources. Thus, even the most low-level processing of beliefs appears to reflect a capacity-limited operation.

Keywords

theory of mind, social cognition, dual-task performance, eye movements, implicit cognitive processes, cognitive load

Received 12/19/11; Accepted 1/15/12

Research Report

Cognitive Load and Implicit Theory of Mind 843

ball to be (Southgate, Senju, & Csibra, 2007; see also Senju, Southgate, Snape, Leonard, & Csibra, 2011).

Do humans fail to understand other individuals’ internal mental states until the age of 4, or is this fundamental ability already present during the 1st year of life? To accommodate these seemingly incongruent findings, Apperly and Butterfill (2009) proposed that throughout the life span, ToM is sub-served by two distinct systems. According to this framework, an earlier-developing system, which operates implicitly (see Schneider, Bayliss, Becker, & Dux, 2011) and is independent of the development of language and executive function (e.g., working memory), is responsible for efficient monitoring of belief-like states. A later-developing system, which is depen-dent on domain-general cognitive functions (e.g., executive function), allows conscious (explicit) ToM inferences. Evi-dence supporting this framework includes a dissociation found in adults with Asperger’s syndrome, who can pass explicit false-belief tasks but do not display eye movement patterns consistent with implicit ToM in a Sally-Anne free-viewing paradigm (Senju, Southgate, White, & Frith, 2009).

Apperly and Butterfill’s (2009) account explains a wide range of data. However, until now, a key test of this theory had yet to be undertaken. That is, no studies had tested whether the implicit ToM system is independent of domain-general, capacity-limited, cognitive resources (e.g., working memory). There is considerable evidence, from both behavioral and neuropsychological studies, that domain-general executive resources contribute strongly to social reasoning in tasks that involve explicit ToM judgments (e.g., McKinnon & Moscovitch, 2007; Rowe, Bullock, Polkey, & Morris, 2001). In the study reported here, we tested the role of such resources in implicit ToM processing by manipulating cognitive load while measuring neurotypical adults’ eye movements in a free-viewing false-belief paradigm modeled after the Sally-Anne task. To ensure that our task tapped implicit ToM, we thor-oughly assessed the extent to which participants engaged in explicit belief processing using an extensive debriefing proce-dure (see also Schneider et al., 2011).

MethodSixty-five neurotypical volunteers (mean age = 20.83 years; 37 females, 28 males) participated in a protocol approved by the University of Queensland’s ethics committee. All com-pleted the Autism-Spectrum Quotient questionnaire (AQ; Baron-Cohen, Wheelwright, Skinner, Martin, & Clubley, 2001), and none scored above the clinical cutoff of 32 (out of 50; mean AQ = 17.14).

Movies portraying scenarios modeled after the Sally-Anne paradigm (see Schneider et al., 2011) were displayed on a 17-in. LCD monitor using Presentation software (Neurobe-havioral Systems, Albany, CA). Participants sat 58 cm from the screen with their head position constrained via a chin rest. Eye movements were measured with an EyeLink 1000 eye tracker (sampling rate: 500 Hz; SR Research, Mississauga,

Ontario, Canada). Filler and experimental movies were pre-sented in a random order over approximately 50 min.

In filler trials, participants saw an actor sitting in a chair behind a desk with two opaque boxes on it. In one type of filler movie, a hand puppet placed a red ball on top of one of the boxes (movie duration = 3 s); in the other type, the puppet placed the red ball in one of the boxes (movie duration = 29 s). The filler movies concluded with a bell sounding and the actor then reaching toward the ball.

There were two types of experimental trials (duration between 66 and 73 s; see Fig. 1a). In false-belief scenarios, which began with the same general scene as the filler movies, the puppet hid the ball in one of the boxes and then moved it into the other box, all while the actor was present and watch-ing. Then the actor left the room, and the puppet moved the ball back to the initial box. This resulted in the actor’s belief mismatching the ball’s actual location when she returned (an example of a false-belief movie is available at http://youtu.be/HMaLIBRwN-Q). The true-belief scenarios were identical to the false-belief trials except that the actor left the room after the puppet first hid the ball (i.e., the actor did not see the ball being moved to the other box and back to the initial box). Thus, upon the actor’s return, her belief was consistent with the ball’s actual location (an example of a true-belief movie is available at http://youtu.be/yf2vVSaaF9Q). The initial (and final) location of the ball was counterbalanced, such that there were two versions of each type of experimental trial (false-belief scenarios with the ball ending up in the box on the right and with the ball ending up in the box on the left; true-belief scenarios with the ball ending up in the box on the right and with the ball ending up in the box on the left).

In each experimental trial, once the actor reentered the room and sat behind the desk, a bell sounded, and the final movie frame froze for approximately 6 s. This frame was divided into three areas of interest (face, left box, and right box) for the eye-tracking analysis. This allowed us to examine our key question: whether participants would view the empty box (no-ball loca-tion) longer when the actor falsely believed the ball was at that location (false-belief condition) than when she correctly believed it was not at that location (true-belief condition). Note that our actor wore a visor to avoid gaze-cuing effects (Frischen, Bayliss, & Tipper, 2007; Schneider et al., 2011). To ensure that the eye movement data reflected implicit ToM processing, we employed a funneled debriefing protocol at the end of each session (as used by Schneider et al., 2011). This protocol, which was adapted from a procedure used to assess implicit higher mental processes in previous work (Bargh & Chartrand, 2000), probed, with increasing specificity, whether participants engaged in conscious processing of the actor’s belief states.

All participants were required to make a simple speeded button press when they saw the actor waving at the puppet (this occurred in one of the types of filler trials, an example of which is available at http://youtu.be/7BkFwInVNcg). These waves occurred in 7 or 15 of the filler trials, depending on condition. This task ensured that participants were motivated

844 Schneider et al.

to watch the movies but not explicitly concerned with the belief state of the actor. Further, it was not meant to severely tax executive processes.

To manipulate cognitive load, we assigned some partici-pants another task to perform concurrently with the movie-viewing task (Fig. 1b). Neither of these tasks was related to belief processing. Participants in the no-load group performed the movie-viewing (wave-detection) task only. Thus, the pro-cedure for this group replicated the method of the first experi-ment in Schneider et al. (2011). Participants in the low- and high-load conditions also listened to a continuous auditory stream of letters randomly selected from the full alphabet and voiced by an English-speaking female (~400 ms per item,

presentation rate = 0.67 Hz); these streams were presented only during the experimental movies (i.e., at times crucial for belief establishment) via headphones adjusted to a comfort-able volume. The onset of each auditory stream was synchro-nized to the start of the movie, and the stream ended just before the bell sounded. Each stream contained two, four, six, eight, or ten 2-back letter repetitions (repetitions separated by one item: e.g., “. . . R, L, R. . .”). Participants in the low-load con-dition were instructed to simply listen to the letters, but not respond to them, as they watched the movies. This condition was designed to tax the executive system to some extent (more than in the no-load condition), as participants needed to avoid being distracted by the continuous auditory stimuli presented

20 30 4010 700 60

False Belief

True Belief

50

a

b

Time (s)

No Load

Low Load

High Load

P Q P X Z N H C I A I T .............

X P Q P Z N H C I A I T .............

0 4 8 12 16 20 24 70

0.4 s1.5 s

“2”

Time (s)

Fig. 1. Illustration of the belief-processing scenarios and cognitive-load manipulation. In experimental trials (a), an actor watched a hand puppet hide a ball in one of two opaque boxes (here, the box on her right side). In the false-belief scenario, the puppet then transferred the ball to the other box, the actor left the room, and the puppet transferred the ball back to the initial box. The returning actor therefore had a false belief about which box contained the ball. In the true-belief scenario, the actor also watched the puppet hide the ball in one of two opaque boxes, but then left the room. Next, the puppet transferred the ball to the other box and then back to the initial box. The returning actor therefore had a true belief about which box contained the ball. In both false-belief and true-belief trials, after the actor reentered the room and was seated (at approximately the 60-s mark), a bell sounded, and the movie was frozen for about 6 s. As part of the movie-viewing task, all participants were required to make a simple speeded button press whenever they saw the actor waving at the puppet (implemented in one type of filler trial). Cognitive load was manipulated by having some participants perform a concurrent primary task (implemented in the experimental trials; b). In the no-load condition, the only task was the movie-viewing task. In both the high- and the low-load conditions, participants also listened to a continuous auditory stream of randomly selected letters; participants in the low-load condition simply listened to the letters, but those in the high-load condition were asked to report the number of 2-back letter repetitions at the end of each trial (here there are two such repetitions, of “P” and “I”).

Cognitive Load and Implicit Theory of Mind 845

during the key phases of belief processing. In the high-load condition, participants were required to count the number of 2-back repetitions and report this total at the end of each experimental trial. This task was designed to draw heavily on executive processes, as participants needed to direct working memory resources toward this task to succeed. The only other instruction given to participants was to watch the movies; thus, the movie component was a free-viewing paradigm with no task related to the belief scenarios.

All participants were presented with 10 false-belief and 10 true-belief trials. In addition, to equate run time for all the groups, we included 40 filler trials (15 of which were wave-detection trials) in the no-load condition and 22 filler trials (7 of which were wave-detection trials) in the low- and high-load conditions. Schneider et al. (2011) have demonstrated that eye movement behavior consistent with implicit ToM processing is identical when these two numbers of filler trials are used.

Results and DiscussionEleven participants were removed from analyses, as debriefing revealed that they may have explicitly processed beliefs. The final sample included 18 participants in each group. Participants in the high-load group performed the 2-back-repetition task with a mean accuracy of 45%, which was significantly above the chance level of 20%, t(17) = 6.91, p < .001, but still quite low. Given this low level of performance, it is clear that this task was demanding, and as participants were required to both maintain and update information, it likely drew heavily on executive resources (Smith & Jonides, 1999).

To assess eye movement behavior, we calculated the per-centage of fixation duration toward each of the three areas of interest (ball, no-ball, and face locations) relative to the total time fixating these three areas in the last 6 s (the final frame) of the experimental movies. These data were submitted to a 3 (group: no load vs. low load vs. high load) × 2 (belief condi-tion: true vs. false belief) × 2 (location: ball vs. no ball) mixed

factorial analysis of variance (ANOVA). Crucially, the three-way interaction was significant, F(2, 51) = 3.61, p = .034, ηp

2 = .124; the pattern of eye movements to the ball and no-ball loca-tions in the true- and false-belief scenarios differed as a func-tion of cognitive load (Fig. 2). To further investigate this interaction, we submitted the data from each group to a sepa-rate 2 (belief condition: true vs. false belief) × 2 (location: ball vs. no ball) repeated measures ANOVA.

For the no-load group, there was a significant two-way interaction, F(1, 17) = 6.95, p = .017, ηp

2 = .290. Planned follow-up t tests revealed that for the no-ball location, the per-centage of fixation duration was higher on false-belief than on true-belief trials, t(17) = 2.32, p = .033; however, no such difference was seen at the ball location (p = .505). There was also an effect of location, with the ball location looked at more overall than the no-ball location, F(1, 17) = 4.91, p = .041, ηp

2 = .224. These results replicate those of Schneider et al. (2011) and demonstrate eye movement behavior consis-tent with belief processing: Participants spent more time look-ing at the no-ball location when the actor believed the ball was at that location (false-belief condition) as opposed to when the actor believed the ball was at the other location (true-belief condition). Recall that participants were not instructed to track the beliefs of the agent and that our debriefing procedure was sensitive enough to detect participants who engaged in explicit ToM analysis. Thus, this belief-tracking behavior appears to have operated implicitly.

Implicit belief processing was not observed in the low-load group, as the two-way interaction between belief condition and location was not significant (p = .174); however, there was a main effect of location, with participants fixating the ball location to a greater extent than the no-ball location overall, F(1, 17) = 5.005, p = .039, ηp

2 = .227. There was no influence of belief condition or location on eye movement behavior in the high-load group, nor did these variables interact (ps > .63). Thus, it appears that increased cognitive load impairs implicit belief processing; the no-load group tracked both the belief

Ball No Ball0

4

8

12

16

20

24

28

Content of Box

Fixa

tion

Dura

tion

(%)

False Belief

True Belief

No Load

Ball No Ball

Content of Box

Low Load

0

4

8

12

16

20

24

28

Ball No Ball

Content of Box

High Load

0

4

8

12

16

20

24

28

Fig. 2. Percentage of fixation duration toward the box containing the ball and toward the box not containing the ball in the false-belief and true-belief conditions, separately for the no-load, low-load, and high-load groups. Error bars represent standard errors of the difference between the true- and false-belief conditions for each location in each group.

846 Schneider et al.

state of the actor and the ball’s location, the low-load group tracked only the actual position of the ball, and the high-load group tracked neither. These results suggest that although there may be a ToM system that operates implicitly, it is not independent of executive function as hypothesized by Apperly and Butterfill (2009).

Did we fail to see eye movement behavior consistent with implicit belief processing in the low- and high-load groups simply because cognitive load impaired visual scene process-ing generally? Our eye movement data suggest this is not the case. First, there was no overall effect of group (F < 1) in the three-way mixed factorial ANOVA; across the three groups, the boxes were fixated for the same percentage of time. Sec-ond, a similar 2 (location: face vs. nonface) × 3 (group) mixed factorial ANOVA comparing the percentage of fixation dura-tion for the face location and the average percentage of fixa-tion duration across the two nonface locations demonstrated only a main effect of location, F(1, 51) = 302.49, p < .001, ηp

2 = .856, with individuals in all groups devoting a greater percentage of fixation duration to the face of the actor (M = 76.56%) than to either of the two boxes (M = 10.27%). The interaction did not approach significance (F < 1). Thus, it appears that cognitive load influenced only the implicit analy-sis of the actor’s belief and the location of the ball. Finally, when we examined eye movement behavior only in the no- and low-load groups, in which we could be sure that partici-pants tracked the ball (given the effect of location), we found a significant Group (no load vs. low load) × Belief Condition (true vs. false belief) × Location (ball vs. no ball) interaction, F(1, 34) = 6.487, p = .016, ηp

2 = .160. Crucially, a follow-up test showed that the only difference between these groups was the relative percentage of fixation duration at the no-ball loca-tion in the false- and true-belief conditions, t(34) = 2.084, p = .045. There was no such effect at the ball location (p = .189). This is further evidence that our main findings do not simply reflect cognitive load disrupting eye movements in general.

Collectively, the present work suggests that, although there may be distinct ToM systems that operate at implicit and explicit levels of processing, both of these appear to draw, at least to some extent, on executive resources. These results stand in contrast to those found for implicit “Level-1” visual perspective calculation (tracking what an agent can or cannot see), a related process, which is not influenced by dual- task manipulations (Qureshi, Apperly, & Samson, 2010). Seemingly, even the most low-level belief analysis reflects a capacity-limited operation. Future work should further exam-ine the relative demands placed on executive resources by explicit and implicit ToM processes.

Acknowledgments

Dana Schneider and Rebecca Lam contributed equally to this work.

Declaration of Conflicting Interests

The authors declared that they had no conflicts of interest with respect to their authorship or the publication of this article.

FundingDana Schneider was supported by a University of Queensland Centennial Scholarship, and Paul E. Dux was supported by an Australian Research Council Discovery Grant and Fellowship (DP0986387).

References

Apperly, I. A., & Butterfill, S. (2009). Do humans have two systems to track beliefs and belief-like states? Psychological Review, 116, 953–970. doi:10.1037/a0016923

Bargh, J. A., & Chartrand, T. L. (2000). The mind in the middle: A practical guide to priming and automaticity research. In H. T. Reis & C. M. Judd (Eds.), Handbook of research methods in social and personality psychology (pp. 253–285). New York, NY: Cambridge University Press.

Baron-Cohen, S. (1995). Mindblindness: An essay on autism and theory of mind. Cambridge, MA: MIT Press.

Baron-Cohen, S., Leslie, A. M., & Frith, U. (1985). Does the autistic child have a “theory of mind”? Cognition, 21, 37–46. doi:10.1016/0010-0277(85)90022-8

Baron-Cohen, S., Wheelwright, S., Skinner, R., Martin, J., & Clubley, E. (2001). The Autism-Spectrum Quotient (AQ): Evidence from Asperger syndrome/high-functioning autism, males and females, scientists and mathematicians. Journal of Autism and Develop-mental Disorders, 31, 5–17. doi:10.1023/A:1005653411471

Call, J., & Tomasello, M. (2008). Does the chimpanzee have a theory if mind? 30 years later. Trends in Cognitive Sciences, 12, 187–192. doi:10.1016/j.tics.2008.02.010

Clements, W. A., & Perner, J. (1994). Implicit understanding of belief. Cognitive Development, 9, 377–395. doi:10.1016/0885-2014(94)90012-4

Frischen, A., Bayliss, A. P., & Tipper, S. P. (2007). Gaze cueing of attention: Visual attention, social cognition and individual differ-ences. Psychological Bulletin, 133, 694–724. doi:10.1037/0033-2909.133.4.694

Frith, U. (2003). Autism: Explaining the enigma (2nd ed.). Oxford, England: Blackwell.

Kovács, Á. M., Téglás, E., & Endress, A. D. (2010). The social sense: Susceptibility to others’ beliefs in human infants and adults. Sci-ence, 330, 1830–1834. doi:10.1126/science.1190792

McKinnon, M. C., & Moscovitch, M. (2007). Domain-general contributions to social reasoning: Theory of mind and deontic reasoning re-explored. Cognition, 102, 179–218. doi:10.1016j .cognition.2005.12.011

Onishi, K., & Baillargeon, R. (2005). Do 15-month-old infants understand false beliefs? Science, 308, 255–258. doi:10.1126/science.1107621

Perner, J., & Lang, B. (1999). Development of theory of mind and executive control. Trends in Cognitive Sciences, 3, 337–344. doi:10.1016/S1364-6613(99)01362-5

Premack, D. G., & Woodruff, G. (1978). Does the chimpanzee have a theory of mind? Behavioral & Brain Sciences, 1, 515–526. doi:10.1017/S0140525X00076512

Qureshi, A. W., Apperly, I. A., & Samson, D. (2010). Executive function is necessary for perspective selection, not Level-1 visual

Cognitive Load and Implicit Theory of Mind 847

perspective calculation: Evidence from a dual-task study of adults. Cognition, 117, 230–236. doi:10.1016/j.cognition.2010 .08.003

Rowe, A. D., Bullock, P. R., Polkey, C. E., & Morris, R. G. (2001). ‘Theory of mind’ impairments and their relationship to execu-tive functioning following frontal lobe excisions. Brain, 124, 600–616. doi:10.1093/brain/124.3.600

Schneider, D., Bayliss, A. P., Becker, S. I., & Dux, P. E. (2011). Eye movements reveal sustained implicit processing of others’ mental states. Journal of Experimental Psychology: General. Advance online publication. doi:10.1037/a0025458

Senju, A., Southgate, V., Snape, C., Leonard, M., & Csibra, G. (2011). Do 18-month-olds really attribute mental states to others? A critical test. Psychological Science, 22, 878–880. doi:10.1177/0956797611411584

Senju, A., Southgate, V., White, S., & Frith, U. (2009). Mind - blind eyes: An absence of spontaneous theory of mind in As- perg er syndrome. Science, 325, 883–885. doi:10.1126/science .1176170

Smith, E. E., & Jonides, J. (1999). Storage and executive processes in the frontal lobes. Science, 283, 1657–1661. doi:10.1126/ science.283.5408.1657

Southgate, V., Senju, A., & Csibra, G. (2007). Action anticipa-tion through attribution of false belief by 2-year-olds. Psycho-logical Science, 18, 587–592. doi:10.1111/j.1467-9280.2007 .01944.x

Wimmer, H., & Perner, J. (1983). Beliefs about beliefs: Representa-tion and constraining function of wrong beliefs in young chil-dren’s understanding of deception. Cognition, 13, 103–128. doi: 10.1016/0010-0277(83)90004-5

http://pss.sagepub.com/Psychological Science

http://pss.sagepub.com/content/17/10/882The online version of this article can be found at:

 DOI: 10.1111/j.1467-9280.2006.01798.x

2006 17: 882Psychological ScienceJung-Beeman

John Kounios, Jennifer L. Frymiare, Edward M. Bowden, Jessica I. Fleck, Karuna Subramaniam, Todd B. Parrish and MarkSudden Insight

The Prepared Mind: Neural Activity Prior to Problem Presentation Predicts Subsequent Solution by  

Published by:

http://www.sagepublications.com

On behalf of: 

  Association for Psychological Science

can be found at:Psychological ScienceAdditional services and information for    

  http://pss.sagepub.com/cgi/alertsEmail Alerts:

 

http://pss.sagepub.com/subscriptionsSubscriptions:  

http://www.sagepub.com/journalsReprints.navReprints:  

http://www.sagepub.com/journalsPermissions.navPermissions:  

What is This? 

- Oct 1, 2006Version of Record >>

at Copenhagen University Library on October 27, 2014pss.sagepub.comDownloaded from at Copenhagen University Library on October 27, 2014pss.sagepub.comDownloaded from

Research Article

The Prepared MindNeural Activity Prior to Problem Presentation PredictsSubsequent Solution by Sudden InsightJohn Kounios,1 Jennifer L. Frymiare,2 Edward M. Bowden,3 Jessica I. Fleck,1 Karuna Subramaniam,3

Todd B. Parrish,3 and Mark Jung-Beeman3

1Department of Psychology, Drexel University; 2Department of Psychology, University of Wisconsin-Madison; and3Department of Psychology and Cognitive Brain Mapping Group, Northwestern University

ABSTRACT—Insight occurs when problem solutions arise

suddenly and seem obviously correct, and is associated

with an ‘‘Aha!’’ experience. Prior theorizing concerning

preparation that facilitates insight focused on solvers’

problem-specific knowledge. We hypothesized that a dis-

tinct type of mental preparation, manifested in a distinct

brain state, would facilitate insight problem solving inde-

pendently of problem-specific knowledge. Consistent with

this hypothesis, neural activity during a preparatory

interval before subjects saw verbal problems predicted

which problems they would subsequently solve with, versus

without, self-reported insight. Specifically, electroenceph-

alographic topography and frequency (Experiment 1) and

functional magnetic resonance imaging signal (Experi-

ment 2) both suggest that mental preparation leading

to insight involves heightened activity in medial frontal

areas associated with cognitive control and in temporal

areas associated with semantic processing. The results

for electroencephalographic topography suggest that non-

insight preparation, in contrast, involves increased

occipital activity consistent with an increase in externally

directed visual attention. Thus, general preparatory

mechanisms modulate problem-solving strategy.

When Louis Pasteur said, ‘‘Chance favors only the prepared

mind,’’ he was likely referring to preparation for the sort of

sudden illumination that enables one to solve a difficult problem

or reinterpret a situation in a new light (Wallas, 1926). Psy-

chologists later called this type of sudden comprehension in-

sight (Smith & Kounios, 1996; Sternberg & Davidson, 1995), a

phenomenon associated with performance on tests of intelli-

gence and creativity (Ansburg & Hill, 2003; Davidson, 1995).

Although insights pop into awareness unexpectedly, or even

unbidden (Kvavilashvili & Mandler, 2004; Metcalfe & Wiebe,

1987; Smith &Kounios, 1996), Pasteur apparently believed that

some form of preparation facilitates insight. One type of prep-

aration involves studying a problem or relevant background

information (Seifert, Meyer, Davidson, Patalano, & Yaniv, 1995;

Wallas, 1926). Such study is obviously helpful, but probably

facilitates problem solving by both insight and noninsight ana-

lytic processing.

We hypothesized another type of preparation, one that does

not depend on information related to specific problems, but that

biases a person toward processing that facilitates solution by

insight. Here, we demonstrate that preparation for problem

solving can be associated with distinct brain states, one biasing

toward solution with insight, the other biasing toward solution

without insight. We examined neural activity associated with

subjects’ preparation immediately prior to the presentation of

each problem and found that the spatial distribution and oscil-

latory frequency of this activity predicts whether the problem

that follows will be solved with insight or noninsight processing,

as marked by the presence or absence of an ‘‘Aha!’’ experience.

Insight has typically been studied by comparing performance

on insight problems, which are often solved with an ‘‘Aha!’’ ex-

perience, with performance on noninsight problems, which are

usually solved without an ‘‘Aha!’’ (Mayer, 1995; Weisberg,

1995). Unfortunately, such classification is not definitive, be-

cause any particular problem could be solved with or without

insight (Bowden, Jung-Beeman, Fleck, & Kounios, 2005). In-

stead, in the present study, we used each subject’s trial-by-trial

judgments of whether each solution became available incre-

mentally or as a sudden insight to classify solutions to individual

Address correspondence to JohnKounios, Department of Psychology,Drexel University, MS 626, 245 N. 15th St., Philadelphia, PA 19102-1192, e-mail: john.kounios@gmail.com, or to Mark Jung-Beeman,Department of Psychology and Cognitive Brain Mapping Group,Northwestern University, 2029 Sheridan Rd., Evanston, IL 60208-2710, e-mail: mjungbee@northwestern.edu.

PSYCHOLOGICAL SCIENCE

882 Volume 17—Number 10Copyright r 2006 Association for Psychological Science at Copenhagen University Library on October 27, 2014pss.sagepub.comDownloaded from

problems as resulting from insight or noninsight processing.

Using this approach, previous studies have demonstrated unique

patterns of behavioral results (Bowden & Jung-Beeman, 2003a)

and neural activity (Jung-Beeman et al., 2004) associated with

insight versus noninsight solutions.

For several reasons, we have inferred that these self-reported

‘‘Aha!’’ experiences reflect the sudden conscious availability of

a solution rather than some ancillary process. For example, (a)

the associated neural activity, a sudden burst of gamma-band

oscillatory activity in the right anterior superior temporal gyrus,

does not reflect subjects’ affective or surprise reactions follow-

ing solutions, because the onset of this activity coincides with,

rather than follows, the conscious availability of the solution; (b)

task-related activity occurs in the same region when people first

start processing a problem, before they experience any solution-

related emotional response (Jung-Beeman et al., 2004); and (c)

this region is a polymodal association area that has not been

implicated in affective or novelty processing (Jung-Beeman,

2005).

In two experiments, using electroencephalography (EEG) and

functional magnetic resonance imaging (fMRI), we assessed

neural activity as people prepared to solve each problem in a

series. We examined activity prior to the presentation of each

problem in order to assess patterns independent of specific

problems and their difficulty. Experiment 1 focused on EEG

power within the alpha (8–13 Hz) frequency band. Alpha, the

brain’s dominant rhythm (Shaw, 2003), reflects cortical deacti-

vation and is inversely related to hemodynamic and metabolic

measures of neural activity (Cook, O’Hara, Uijtdehaage, Man-

delkern, & Leuchter, 1998; Cooper, Croft, Dominey, Burgess, &

Gruzelier, 2003; Goldman, Stern, Engel, & Cohen, 2002; Laufs

et al., 2003; Ray&Cole, 1985;Worden, Foxe,Wang, & Simpson,

2000). The topographic distribution of alpha across the scalp

therefore mirrors the spatial distribution of neural activity

(Pfurtscheller & Lopes da Silva, 1999). We focused on the low-

alpha frequency band (8–10Hz), because high alpha (10–13Hz)

is typically dominated by an occipital alpha rhythm reflecting

gating of visual information (Gevins & Smith, 2000). Effects

were also found in the gamma band (> 30 Hz), but could not be

reliably distinguished from electromyograph artifact, so they are

not discussed in this report.

EXPERIMENT 1

Method

After giving informed consent, 19 subjects attempted to solve

186 problems. On each trial (Fig. 1), they indicated by bimanual

button press that they were prepared to begin working on a

problem, thereby initiating the display of a visual fixation mark.

This button press was the midpoint of the 2-s epoch selected as

the preparation interval of interest. After 1 s, this fixation mark

was replaced by the three words of a compound remote-associates

problem. For each problem (e.g., pine, crab, sauce), subjects

attempted to produce a solution word (e.g., apple) that could be

combined with each of the three problem words to form a com-

mon compound or phrase (pineapple, crabapple, applesauce).

These problems were used because a large number are avail-

able, they can be solved relatively quickly, and similar problems

have been used successfully in numerous studies of insight and

creativity (for review, see Bowden & Jung-Beeman, 2003b).

Also, they can be solved either with insight or with noninsight

analytic processes (Bowden & Jung-Beeman, 2003a; Jung-

Beeman et al., 2004), so we could compare these two general

strategies while holding task and problem type constant. Pre-

vious studies have shown that subjects solving such problems

tend to use each of these strategies about half the time (Bowden

et al., 2005).

If subjects achieved solution, they made an immediate bi-

manual button press indicating that they had solved the prob-

lem, verbalized the solution when prompted, and then, when

prompted, pressed one of two buttons to indicate whether or not

the solution had been achieved by insight. (Prior instructions to

subjects explained the notion of insight as sudden awareness of

the solution—Jung-Beeman et al., 2004). After a 2-s intertrial

Fig. 1. Time line of events on a trial in Experiment 1. A ‘‘Ready?’’ message was displayed on a monitor, and the subject made abimanual button press (‘‘S’’) when he or she was prepared to start the trial and view the three words constituting a problem. Theproblemwas displayed after a 1-s visual fixationmark. The subject respondedwith another button press (‘‘R’’) as soon as he or shehad solved the problem.This initiated a prompt to verbalize the solution (in this case, goose), followedby another prompt tomake abutton press to indicate whether the verbalized solution was accompanied by an experience of insight. This report presentselectroencephalography (EEG) results for the bracketed preparatory interval consisting of the 2 s prior to the display of theproblem.

Volume 17—Number 10 883

J. Kounios et al.

at Copenhagen University Library on October 27, 2014pss.sagepub.comDownloaded from

interval, a ‘‘Ready?’’ prompt appeared; when ready, subjects

initiated the next trial with a bimanual button press.

High-density (128-channel) EEG was continuously recorded

(referenced to digitally linked mastoids) at 250 Hz (0.02–100

Hz). Eyeblink artifacts were removed using EMSE 5.0 (Source

Signal Imaging, Inc., www.sourcesignal.com). Trials containing

other artifacts were identified by visual inspection and deleted.

EEG segments corresponding to the preparation intervals were

extracted and, on the basis of subjects’ performance on the

following problem, were sorted into three types of preparation:

preparation leading to solution with insight, preparation leading

to solution without insight, or time-out (no response within 30 s).

(Errors were rare, and trials with errors were deleted.) EEG

power was estimated by computing power spectra on these

segments. The University of Pennsylvania’s institutional review

board approved the study.

Results

Participants solved 46.2% (SD5 8.2) of the problems correctly

within the 30-s time limit. Subjects labeled 56.2% (SD5 8.3) of

solutions as insight solutions (average median response time5

7.70 s, SD5 2.60) and 42.5% (SD5 8.9) as noninsight solutions

(7.48 s, SD5 3.08). Of all responses, 11.5% (SD5 11.3) were

errors.

As predicted, distinct alpha topographies were associated

with preparation for solving problems with insight versus non-

insight processing. Moreover, visual inspection of power spectra

suggested that insight preparation and noninsight preparation

were associated with different peak oscillatory frequencies

within the low-alpha frequency band. These observations were

substantiated by two sets of analyses.

The first set contrasted insight preparation against prepar-

ation that led to time-outs, and separately contrasted noninsight

preparation against time-out preparation. These analyses also

allowed us to assess whether preparation preceding successful

problem solving differed from preparation preceding unsuc-

cessful problem processing. Two repeated measures analyses of

variance (ANOVAs) were performed on log-transformed EEG

power to examine anterior-posterior and hemispheric differ-

ences in two low-alpha subbands (the frequency factor: 8–9 Hz

vs. 9–10 Hz, determined by visual inspection of the power

spectra) for insight, noninsight, and time-out trials (the trial-

type factor). The first ANOVA examined power at left and right

anterior-frontal (AF1/2) and left and right occipital (O9/10) elec-

trodes (allowing comparison of scalp topographies). These elec-

trodes were chosen a priori in order to maximize distance

between electrodes, thereby minimizing the influence of EEG

volume conduction. Topographic factors with two levels were uti-

lized in the initial ANOVA to avoid reduction in statistical power

associated with correction for nonsphericity (Dien & Santuzzi,

2005). Relevant significant effects included a Frequency �Trial Type � Anterior-Posterior interaction, F(2, 36) 5 3.95,

prep 5 .93, e2 5 .18. A second ANOVA utilized electrodes with

more lateral placements (left and right frontal, F7/8; parietal,

P7/8; and temporal, T7/8) in order to assess possible hemi-

spheric effects; this analysis yielded a significant Trial Type �Hemisphere interaction, F(2, 36) 5 4.07, prep 5 .93, e2 5 .18.

To specify the topographic differences driving the interactions

in the omnibus ANOVAs, we computed the statistical parametric

maps shown in Figure 2a as follow-up tests. The use of these

follow-up tests to specify effects driving the interaction is

justified by the presence of the interaction in the overall

Fig. 2. Results from Experiment 1: alpha-band electroencephalographic(EEG) topography during the 2-s preparatory interval before the problemwas displayed. Plotted values are t scores of electrode-by-electrode com-parisons. The maps in (a) show results for comparisons between unsolvedproblems (time-outs, or TOs) and problems solved with insight processing(I) or with noninsight processing (NI); the left column shows maps forcomparisons in the 8- to 9-Hz frequency band, and the right column showsmaps for comparisons in the 9- to 10-Hz band. Red and orange regionsindicate electrode sites at which I or NI trials exhibited less alpha power(i.e., more neural activity) than TO trials; the middle 66% of the colorscale is grayed out. The maps in (b) show comparisons between insightpreparation at peak power (9–10 Hz) and noninsight preparation at peakpower (8–9Hz; insight preparationminus noninsight preparation). Yellowregions show electrode sites at which insight preparation exhibited lessalpha power than noninsight preparation. Blue regions show electrodesites at which noninsight preparation exhibited less alpha power than in-sight preparation. The middle 33% of the color scale is grayed out.

884 Volume 17—Number 10

Neural Preparation for Insight

at Copenhagen University Library on October 27, 2014pss.sagepub.comDownloaded from

ANOVA.1 Compared with time-out preparation, insight prep-

aration was associated with greater neural activity (i.e., less

alpha power) peaking over midfrontal cortex (9–10 Hz; Fig. 2a,

top right) and left anterior-temporal cortex (8–9Hz and 9–10Hz;

Fig. 2a, top row). In contrast, noninsight preparation, compared

with time-out preparation, was associated with greater neural

activity (decreased 8- to 9-Hz alpha; Fig. 2a, bottom left)

peaking over occipital cortex.

Notably, insight and noninsight preparation (i.e., the two

forms of successful preparation) showed no common differences

from time-out (i.e., unsuccessful) preparation. This suggests that

subjects did not fail to prepare on some trials (e.g., by not at-

tending); rather, they failed on some trials because they could

not solve those particular problems (quickly enough). They

likely engaged in more insight processing than noninsight pro-

cessing on some unsuccessful trials and in more noninsight

processing than insight processing on others, though we have no

way of sorting those trials.

We explored the data further in a set of more focused analyses.

An ANOVA compared power values for insight trials (in the 9- to

10-Hz range) and noninsight trials (in the 8- to 9-Hz range) at

left and right anterior-frontal (AF1/2) and occipital (O9/10)

electrodes. Direct comparison of the topography correspond-

ing to the peak low-alpha frequency associated with insight

preparation against the topography corresponding to the peak

low-alpha frequency associated with noninsight preparation

optimized the spatial resolution of the comparison. Relevant

findings included a significant Insight � Anterior-Posterior

interaction, F(1, 18) 5 8.46, prep 5 .97, e2 5 .32. There were

also marginally significant Insight � Hemisphere, F(1, 18) 5

4.04, prep 5 .91, e2 5 .18, and Insight � Anterior-Posterior �Hemisphere, F(1, 18)5 3.04, prep5 .88, e25 .14, interactions.

To specify the topographic differences underlying these inter-

actions, we computed a statistical parametric map as a follow-up

test (Fig. 2b). Direct comparison of peak low-alpha topographies

associated with insight preparation (9–10 Hz) versus noninsight

preparation (8–9 Hz) showed greater neural activity (i.e., less

alpha) associated with insight preparation peaking over mid-

frontal, left temporal, right temporal, right inferior frontal, and

bilateral somatosensory cortex and greater activity associated

with noninsight preparation over a broad region of posterior

cortex (Fig. 2b). (The somatosensory activation likely reflects

activity associated with the bimanual button press subjects

made to initiate a trial.)

It is possible that a subject’s brain state changes slowly over

the course of an experimental session, because of practice or

fatigue. However, the proportion of problems solved by insight

did not change from the first to the second half of the session

(F< 1). Another possibility is that a subject’s brain state varies

on a shorter time scale, perhaps over the course of a few trials,

entering an insight or a noninsight mode for some time before

changing state. This would result in temporal clustering of trial

types. However, temporal clustering of trial types in our data did

not differ from that of simulated random data (all Fs < 1.0).

These results suggest that activity associated with insight versus

noninsight processing changed within the few seconds between

trials. Given that subjects initiated the presentation of each

problem when they felt prepared, these trial-by-trial changes in

neural activity likely reflect differential preparation prior to the

presentation of individual problems.

EXPERIMENT 2

EEG allowed us to isolate the preparatory interval with high

precision, but provided less precise spatial information about

the relevant brain regions. In Experiment 2, fMRI confirmed and

specified the brain regions involved in preparation, though with

less precise isolation of the preparatory interval.

Method

After giving informed consent, 25 subjects attempted to solve

135 compound remote-associate problems during fMRI scan-

ning; the time limit for each problem was 15 s. Five subjects

were replaced: Four showed excessive movement or poor MRI

signal, and one responded ‘‘noninsight’’ on only two trials.

The paradigm described for Experiment 1 was modified

slightly to optimize fMRI data acquisition. The preparation

interval preceding each problem comprised a rest period (with a

fixation cross) of varying length (2, 4, 6, or 8 s, randomly chosen)

that followed the insight judgment (after a solved problem) or the

time-out event (after an unsolved problem). No button press was

required to initiate a trial, and subjects could not predict the

length of the rest period, so the preparation was likely somewhat

more passive than in Experiment 1.

Scanning was performed at Northwestern University’s Center

for Advanced MRI using a 3-T Siemens Trio scanner with

standard head coil. Head motion was restricted with plastic

calipers built into the coil and a vacuum pillow. Anatomical

high-resolution T1-weighted images were acquired in the axial

plane at the end of every session. In-plane functional images

were acquired using a gradient echo-planar sequence (time to

repetition, TR5 2 s for 38 slices that were 3 mm thick, time to

echo5 20 ms, matrix size of 64� 64 in a 220-mm field of view).

Each of five runs began with an 8-s saturation period; then

participants solved problems for up to 10 min, 20 s (the final run

was truncated when subjects finished solving problems).

Functional and anatomical images were co-registered through

time, spatially smoothed with a 7.5-mm Gaussian kernel, and fit

to a common template. The data were analyzed using general

1Topographic ANOVAs provide relatively sensitive statistical tests becausethey pool variance across electrodes, but they provide relatively coarse topo-graphic information. In contrast, t-score mapping provides more specific topo-graphic information about the effects driving ANOVA results, though with lessstatistical power because variance is not pooled across electrodes.

Volume 17—Number 10 885

J. Kounios et al.

at Copenhagen University Library on October 27, 2014pss.sagepub.comDownloaded from

linear model analysis that extracted average estimated response

to each trial type, correcting for linear drift and removing signal

changes correlated with head motion. For each participant, an

event-related analysis contrasted fMRI signal for insight versus

noninsight preparation intervals (for 4 TRs reflecting 4.1–12.0 s

following the onset of the preparation period). Between-subjects

consistency in difference scores (insight preparation minus

noninsight preparation) was analyzed in a second-stage random-

effects analysis. The significance threshold combined cluster

size and t values for every voxel within a cluster: Clusters ex-

ceeded 1,000 mm3 in volume, with each voxel reliably different

across participants, t(24)5 3.374, p< .0025, uncorrected. This

combination threshold yields very low false-positive rates in

simulations.

Signal acquisition and initial data analyses for this experi-

ment are described in detail elsewhere (Virtue, Haberman,

Clancy, Parrish, & Jung-Beeman, in press). Northwestern Uni-

versity’s institutional review board approved the study.

Results

Participants solved 47% (SD 5 9.9) of the problems correctly

within the 15-s time limit and labeled 55.8% (SD 5 13.9) of

solutions as insight solutions (median response time 5 5.84 s,

SD 5 1.24) and 43.4% (SD 5 13.6) as noninsight solutions

(median response time 5 7.81 s, SD 5 1.79). Of all responses,

6.6% (SD 5 5.4) were errors.

The fMRI signal changed in several brain areas during the

preparation interval. Most areas showed decreasing signal

during preparation, as neural activity returned to baseline. A

few areas showed increased signal, indicating increasing neural

activity during ‘‘rest.’’ This increase was strongest in anterior

cingulate cortex (ACC). Small regions within posterior cingulate

cortex (PCC) and bilateral posterior middle and superior tem-

poral gyri (M/STG) also showed slightly increasing or sustained

activity during preparation.

Of greater interest is that signal change during the preparation

interval varied systematically according to whether the subse-

quent problem was solved with insight versus noninsight (Fig. 3,

Table 1). The fMRI results essentially replicate those observed

with EEG in Experiment 1. In ACC, preparation that preceded

solutions with insight increased fMRI signal more than did

preparation that preceded solutions without insight. In PCC and

bilateral posterior M/STG, signal was stronger for insight than

noninsight preparation, mostly because signal decreased more

for noninsight than insight preparation. Both significant tem-

poral clusters appeared to comprise several smaller clusters (a

small subregion in each hemisphere showed greater signal in-

crease for insight preparation; the remainder of each cluster

showed greater signal decrease for noninsight preparation).

There were also small, nonsignificant clusters in the anterior-

temporal region. No clusters showing the opposite effect ex-

ceeded our combined significance threshold. However, at lower

t thresholds, the largest cluster (375 mm3 at p< .01, 1,344 mm3

at p < .05) showing stronger signal for noninsight than insight

preparation was in left middle and inferior occipital cortex

(Table 1).

GENERAL DISCUSSION

The observed effects on EEG topographies, peak low-alpha

frequencies, and fMRI signal all demonstrate that the neuronal

populations active prior to the presentation of problems subse-

quently solved with insight are different from the neuronal

populations active prior to the presentation of problems subse-

quently solved with noninsight processing. Note that differences

in brain activity during preparation were not influenced by the

specific content of the problems, which were not displayed until

after the epoch examined. Thus, subjects engaged distinct

patterns of mental preparation.

Although the mere demonstration of such preparatory states is

important, the anatomical pattern evident across the two ex-

periments also suggests specific preparatory mechanisms

facilitating insight. The convergence across methods is critical,

as the fMRI data, providing specific anatomical information,

replicate the effects of insight preparation observed with EEG.2

We begin with the ACC region identified with fMRI. This is the

likely source of the insight-related midfrontal activity observed

with EEG.3 Beyond showing an insight-noninsight difference,

this region showed the most robust pattern of increasing fMRI

signal during the preparation period, that is, during ‘‘rest.’’ ACC

has been associated with monitoring for competition among

potential responses or processes. Such conflict monitoring can

signal the need for top-down cognitive control mechanisms fa-

cilitating the maintenance or switching of attentional focus, or

selection from competing responses (Badre & Wagner, 2004;

Botvinick, Cohen, & Carter, 2004; Kerns et al., 2004; Miller &

Cohen, 2001). According to this interpretation, increased ACC

activity is followed by increased top-down control. Notably,

ACC activity increased during the preparation period, when no

obvious response conflict existed. Recent studies suggest a

possible reason: ACCmay be involved in suppressing irrelevant

thoughts (Anderson et al., 2004; Wyland, Kelley, Macrae, Gor-

don, & Heatherton, 2003), such as daydreams or thoughts re-

lated to a preceding event (in this case, the preceding trial).

Thus, ACC activity may have allowed participants to attack the

next problemwith a ‘‘clean slate.’’ This explanation assumes that

insight processing is more susceptible to internal interference

2We did not contrast insight or noninsight preparation against time-outpreparation in Experiment 2 (as we did in Experiment 1), because the contrastwould have been confounded by some temporally proximal solutions, whichwould have been absent in the time-out trials, and which would have affectedfMRI signal.

3Most prior studies have associated ACC activity with changes in theta-bandEEG power (Luu, Tucker, & Makeig, 2004), but some studies have associated itwith alpha-band oscillations (Goldman et al., 2002).

886 Volume 17—Number 10

Neural Preparation for Insight

at Copenhagen University Library on October 27, 2014pss.sagepub.comDownloaded from

than is noninsight processing (Schooler, Ohlsson, & Brooks,

1993), thereby necessitating greater suppression of extraneous

thoughts.

Thus, in the context of problem solving, the activity observed

in ACC prior to insight may reflect increased readiness to

monitor for competing responses, and to apply cognitive control

mechanisms as needed to (a) suppress extraneous thoughts, (b)

initially select prepotent solution spaces or strategies, and, if

these prove ineffective, (c) subsequently shift attention to a

nonprepotent solution or strategy. Such shifts are characteristic

of insight.

We speculate that during insight preparation, cognitive con-

trol mechanisms were modulating activity in brain areas related

to semantic processing. Greater neural activity was observed for

insight than for noninsight preparation in bilateral temporal

cortex (with this activity being more extensive on the left than on

the right, in both experiments). We propose that this temporal-

lobe activity reflects preparation for semantic activation and

Fig. 3. Results from Experiment 2: structural images averaged across all 25 subjects, showing all voxels with stronger functionalmagnetic resonance imaging (fMRI) signal, t(24)5 3.375, p < .0025, uncorrected, for preparation periods that led to insight solutionsthan for preparation periods that led to noninsight solutions. Each row shows (left to right) axial, sagittal, and coronal images (with theleft hemisphere on the left of axial and coronal images) centered on clusters (circled) of significant size (i.e., larger than 1,000 mm3; noclusters of significant size showed the reverse pattern). For each circled region, the graph on the right shows the average percentagesignal change during the shaded 6- to 12-s interval associated with neural activity during the 2- to 8-s preparatory interval (the offsetbetween these intervals being due to the lag of the hemodynamic response). The blue line shows signal related to insight (I) preparation,the pink line shows signal related to noninsight (NI) preparation (both with standard error bars), and the green line shows the sub-traction (I � NI). Analyses identifying significant voxels tested the contrast within the four TRs (times to repetition) highlighted in theyellow shaded region (i.e., TR ending at 6 s throughTRending at 12 s), which corresponds to the expectedpeak signal for the preparationperiod. (The preceding button press elicited peak signal inmotor cortex at 4 s in these graphs.)Results are shown for four clusters: (a) leftposteriormiddle/superior temporal gyri (L. PostM/STG), (b) anterior cingulate cortex (ACC), (c) posterior cingulate cortex (PCC), and(d) right posterior middle/superior temporal gyri (R. Post M/STG). Ant 5 anterior; Post 5 posterior; L 5 left; R 5 right.

Volume 17—Number 10 887

J. Kounios et al.

at Copenhagen University Library on October 27, 2014pss.sagepub.comDownloaded from

results from top-down processes such as those associated with

the ACC. Within a theoretical framework of bilateral semantic

processing (Jung-Beeman, 2005; also Faust & Lavidor, 2003)

partly based on cytoarchitectonic studies (Hutsler & Galuske,

2003), the bilateral (or slightly leftward) distribution of this

activity indicates that subjects were prepared to retrieve both

prepotent associations (predominantly in the left posterior

M/STG) and weaker associations (in the right posterior M/STG).

This hypothesis is also consistent with a recently proposed

model of the neural basis of insight in verbal problem solving

(Bowden et al., 2005; Jung-Beeman et al., 2004) according to

which solvers initially focus on prepotent associations, possibly

reaching an impasse if this does not lead to a solution. However,

solvers may maintain weak activation for the solution (Beeman

& Bowden, 2000) due to coarser semantic coding in the right

hemisphere (Bowden&Beeman, 1998; Bowden& Jung-Beeman,

2003a). Solvers may overcome impasse by switching attention to

this weakly activated representation, suddenly increasing its

strength. This shift of attention to a nonprepotent solution (or

solving process) involves cognitive control mechanisms such as

those associated with the ACC. This shift of attention may be

less likely to occur when people use a less controlled mode of

processing based on passive accrual of evidence.

This interpretation is consistent with a sudden increase in

high-frequency EEG power (i.e., gamma-band oscillations

peaking at 40 Hz) observed to occur about 0.3 s before insight

(compared with noninsight) solutions (Jung-Beeman et al.,

2004). This burst of gamma-band activity was focused at right

anterior superior-temporal electrodes and corresponded to

activity detected by fMRI in the underlying cortex (with no

significant insight effect in the left anterior temporal lobe). This

right anterior temporal activity may reflect the sudden emer-

gence into consciousness of the correct solution (Jung-Beeman

et al., 2004).

The current results extend this prior work by demonstrating

that this insight response is the culmination of mechanisms that

begin even before a problem is presented. Insight preparation

modulates processing and biases toward insight solutions by

increasing readiness in (a) left posterior temporal cortex, whose

activation is hypothesized to reflect readiness to initially pursue

prepotent associations, and (b) ACC, whose activation is hy-

pothesized to reflect cognitive control mechanisms that facilitate

initially attending to prepotent associations, then discretely

shifting attention to nonprepotent associations—which may

include solution-related information activated in the right an-

terior temporal region.

One additional area, PCC, showed stronger fMRI signal for

insight than for noninsight preparation, perhaps reflecting at-

tentional differences (Small et al., 2003). No corresponding ef-

fect was evident in the EEG data. It is possible, however, that an

effect was present in this deep area, but was canceled out by the

opposite effect measured over posterior cortex, which was likely

generated in more superficial brain areas. Specifically, EEG

revealed greater occipital-parietal activity during noninsight

than during insight preparation; therewas a similar, though not sig-

nificant, effect in the fMRI results. It is possible that this effect

was stronger in the EEG experiment because the preparation

period was more active and predictable; subjects pressed a

button to indicate readiness in the EEG experiment, but not in

the fMRI experiment. This posterior cortical activity during

TABLE 1

Brain Areas Showing Significantly Different Functional Magnetic Resonance Imaging Signal for Insight

Preparation Versus Noninsight Preparation

Gyrus or structureBrodmann

AreaVolume(mm3)

Centercoordinates

Signalchange (%)

Mean t Max tEffectsize (d)x y z Mean Max

Insight preparation > noninsight preparation

Left posterior M/STG 39, 37, 22 2,031 �49 �62 15 0.06 0.08 3.6 4.7 0.98

Anterior cingulate 32, 24 1,922 �4 41 8 0.08 0.10 3.8 4.8 0.94

Posterior cingulate 31 1,594 �3 �47 32 0.08 0.11 4.0 5.6 0.96

Right posterior M/STG 37, 39, 22 1,266 52 �62 9 0.07 0.10 3.7 4.8 1.00

Left amygdala — 438 �25 �7 �9 0.07 0.10 3.8 4.6 —

Left middle temporal gyrus 21 438 �62 �32 �6 0.07 0.10 3.7 4.6 —

Noninsight preparation > insight preparationa

Left middle and inferior

occipital gyrus 18 375 �30 �98 2 �0.09 �0.12 �3.1 �3.7 —

Note. We used a strict threshold for significance, requiring a cluster size of 1,000 mm3 and requiring that all voxels show aconsistent effect across subjects, t(24)5 3.374, p< .0025. For thoroughness, all clusters down to 300 mm3 are listed. For eachcluster listed, we report the signal difference between insight and noninsight preparation as a percentage of the average signalwithin the cluster, as well as the average andmaximum t score for all voxels within the cluster. For significant clusters, we includethe cluster-wise effect size, d. M/STG 5 middle and superior temporal gyri.aNo clusters showed significantly greater signal for noninsight preparation than for insight preparation, so we list the largestarea at a lower threshold, 375 mm3 at t(24) 5 2.795, p < .01.

888 Volume 17—Number 10

Neural Preparation for Insight

at Copenhagen University Library on October 27, 2014pss.sagepub.comDownloaded from

noninsight preparation may suggest readiness to pursue a less

controlled, bottom-up mode of processing, for example, by in-

creasing visual attention just before the problem is displayed.

CONCLUSIONS

The present study demonstrates that a person’s preparatory

brain state even prior to seeing a problem influences whether the

person will solve that problem with insight or noninsight pro-

cessing. Insight preparation could be characterized as preparing

to strongly activate prepotent candidate solutions while also

preparing to switch attention to nonprepotent candidates,

thereby enabling retrieval of weakly activated solutions char-

acterized by remote associations among problem elements. In

contrast, noninsight preparation could be characterized as ex-

ternal attentional focus on the source of the imminent problem.

The fact that subjects use both of these forms of preparation

suggests either that they spontaneously alternate strategies or

that one form of preparation, presumably insight preparation

with its top-down component, is perhaps too cognitively de-

manding to use for every problem in a series.

Future studies will further specify these preparation-related

brain states and their determinants. Ideally, this line of research

could lead to the development of techniques for facilitating or

suppressing insight in order to optimize performance for dif-

ferent types of problems and contexts. In sum, this work may

show how to lessen the impact of chance on efforts to reach

insightful solutions, a goal that Pasteur would likely have en-

dorsed.

Acknowledgments—Grants DC-04818 (to J.K.) and DC-04052

(to M.J.-B.) from the National Institutes of Deafness and Other

Communication Disorders supported this research. We thank D.

Stephen Lindsay, James Cutting, and an anonymous reviewer for

helpful comments.

REFERENCES

Anderson, M.C., Ochsner, K.N., Kuhl, B., Cooper, J., Robertson, E.,

Gabrieli, S.W., Glover, G.H., & Gabrieli, J.D.E. (2004). Neural

systems underlying the suppression of unwanted memories. Sci-ence, 303, 232–235.

Ansburg, P.I., & Hill, K. (2003). Creative and analytic thinkers differ in

their use of attentional resources. Personality and IndividualDifferences, 34, 1141–1152.

Badre, D., & Wagner, A.D. (2004). Selection, integration, and conflict

monitoring: Assessing the nature and generality of prefrontal

cognitive control mechanisms. Neuron, 41, 473–487.Beeman, M.J., & Bowden, E.M. (2000). The right hemisphere maintains

solution-related activation for yet-to-be solved insight problems.

Memory & Cognition, 28, 1231–1241.Botvinick, M.M., Cohen, J.D., & Carter, C.S. (2004). Conflict moni-

toring and anterior cingulate cortex: An update. Trends in Cog-nitive Sciences, 8, 539–546.

Bowden, E.M., & Beeman, M.J. (1998). Getting the right idea: Semantic

activation in the right hemisphere may help solve insight prob-

lems. Psychological Science, 9, 435–440.Bowden, E.M., & Jung-Beeman, M. (2003a). Aha! Insight experience

correlates with solution activation in the right hemisphere. Psy-

chonomic Bulletin & Review, 10, 730–737.Bowden, E.M., & Jung-Beeman, M. (2003b). Normative data for 144

compound remote associate problems. Behavior Research Meth-

ods, Instruments, & Computers, 35, 634–639.Bowden, E.M., Jung-Beeman, M., Fleck, J., & Kounios, J. (2005). New

approaches to demystifying insight. Trends in Cognitive Sciences,

9, 322–328.Cook, I.A., O’Hara, R., Uijtdehaage, S.H., Mandelkern,M., & Leuchter,

A.F. (1998). Assessing the accuracy of topographic EEG mapping

for determining local brain function. Electroencephalography and

Clinical Neurophysiology, 107, 408–414.Cooper, N.R., Croft, R.J., Dominey, S.J.J., Burgess, A.P., & Gruzelier,

J.H. (2003). Paradox lost? Exploring the role of alpha oscillations

during externally vs. internally directed attention and the impli-

cations for idling and inhibition hypotheses. International Journal

of Psychophysiology, 47, 65–74.Davidson, J.E. (1995). The suddenness of insight. In R.J. Sternberg &

J.E. Davidson (Eds.), The nature of insight (pp. 125–155). Cam-

bridge, MA: MIT Press.

Dien, J., & Santuzzi, A.M. (2005). Application of repeated measures

ANOVA to high-density ERP datasets: A review and tutorial. In

T.C. Handy (Ed.), Event-related potentials: A methods handbook

(pp. 57–82). Cambridge, MA: MIT Press.

Faust, M., & Lavidor, M. (2003). Semantically convergent and se-

mantically divergent priming in the cerebral hemispheres: Lexical

decision and semantic judgment. Cognitive Brain Research, 17,

585–597.

Gevins, A., & Smith, M.E. (2000). Neurophysiological measures of

working memory and individual differences in cognitive ability

and cognitive style. Cerebral Cortex, 10, 829–839.Goldman, R.I., Stern, J.M., Engel, J., & Cohen, M.S. (2002). Simul-

taneous EEG and fMRI of the alpha rhythm. NeuroReport, 13,

2487–2492.

Hutsler, J., & Galuske, A.W. (2003). Hemispheric asymmetries in

cerebral cortical networks. Trends in Neurosciences, 26, 429–435.Jung-Beeman, M. (2005). Bilateral brain processes for comprehending

natural language. Trends in Cognitive Sciences, 9, 512–518.Jung-Beeman, M., Bowden, E.M., Haberman, J., Frymiare, J.L.,

Arambel-Liu, S., Greenblatt, R., Reber, P.J., & Kounios, J. (2004).

Neural activity when people solve verbal problems with insight.

Public Library of Science Biology, 2, E97.Kerns, J.G., Cohen, J.D., MacDonald, A.W., III, Cho, R.Y., Stenger,

V.A., & Carter, C.S. (2004). Anterior cingulate conflict monitoring

and adjustments in control. Science, 303, 1023–1026.Kvavilashvili, L., & Mandler, G. (2004). Out of one’s mind: A study of

involuntary semantic memories. Cognitive Psychology, 48, 47–94.Laufs, H., Krakow, K., Sterzer, P., Eger, E., Beyerle, A., & Salek-

Haddadi, A. (2003). Electroencephalographic signatures of atten-

tional and cognitive default modes in spontaneous brain activity

fluctuations at rest. Proceedings of the National Academy of Sci-

ences, USA, 100, 11053–11058.Luu, P., Tucker, D.M., & Makeig, S. (2004). Frontal midline theta and

the error-related negativity: Neurophysiological mechanisms of

action regulation. Clinical Neurophysiology, 115, 1821–1835.Mayer, R.E. (1995). The search for insight: Grappling with Gestalt

psychology’s unanswered questions. In R.J. Sternberg & J.E.

Volume 17—Number 10 889

J. Kounios et al.

at Copenhagen University Library on October 27, 2014pss.sagepub.comDownloaded from

Davidson (Eds.), The nature of insight (pp. 3–32). Cambridge, MA:

MIT Press.

Metcalfe, J., & Wiebe, D. (1987). Intuition in insight and noninsight

problem solving. Memory & Cognition, 15, 238–246.Miller, E.K., & Cohen, J.D. (2001). An integrative theory of prefrontal

cortex function. Annual Review of Neuroscience, 24, 167–202.Pfurtscheller, G., & Lopes da Silva, F.H. (1999). Handbook of electro-

encephalography and clinical neurophysiology–revised series: Vol.6. Event-related desynchronization. Amsterdam: Elsevier.

Ray, W.J., & Cole, H.W. (1985). EEG alpha activity reflects attentional

demands, and beta activity reflects emotional and cognitive pro-

cesses. Science, 228, 750–752.Schooler, J.W., Ohlsson, S., & Brooks, K. (1993). Thoughts beyond

words: When language overshadows insight. Journal of Experi-mental Psychology: General, 122, 166–183.

Seifert, C.M., Meyer, D.E., Davidson, N., Patalano, A.L., & Yaniv, I.

(1995). Demystification of cognitive insight: Opportunistic as-

similation and the prepared-mind perspective. In R.J. Sternberg&

J.E. Davidson (Eds.), The nature of insight (pp. 65–124). Cam-bridge, MA: MIT Press.

Shaw, J.C. (2003). The brain’s alpha rhythms and the mind. Amsterdam:Elsevier.

Small, D.M., Gitelman, D.R., Gregory, M.D., Nobre, A.C., Parrish, T.B.,

& Mesulam, M.-M. (2003). The posterior cingulate and medial

prefrontal cortex mediate the anticipatory allocation of spatial

attention. NeuroImage, 18, 633–641.

Smith, R.W., & Kounios, J. (1996). Sudden insight: All or none pro-

cessing revealed by speed-accuracy decomposition. Journal ofExperimental Psychology: Learning, Memory, and Cognition, 22,1443–1462.

Sternberg, R.J., & Davidson, J.E. (Eds.). (1995). The nature of insight.Cambridge, MA: MIT Press.

Virtue, S., Haberman, J., Clancy, Z., Parrish, T., & Jung-Beeman, M. (in

press). Neural activity during inference generation. CognitiveBrain Research.

Wallas, G. (1926). The art of thought. New York: Harcourt Brace Jo-

vanovich.

Weisberg, R.W. (1995). Prolegomena to theories of insight in problem

solving: A taxonomy of problems. In R.J. Sternberg & J.E. Da-

vidson (Eds.), The nature of insight (pp. 157–196). Cambridge,MA: MIT Press.

Worden, M.S., Foxe, J.J., Wang, N., & Simpson, G.V. (2000). Antici-

patory biasing of visuospatial attention indexed by retinotopically

specific alpha-band electroencephalography increases over oc-

cipital cortex. Journal of Neuroscience, 20, RC63.Wyland, C.L., Kelley,W.M., Macrae, C.N., Gordon, H.L., &Heatherton,

T.F. (2003). Neural correlates of thought suppression. Neuro-psychologia, 41, 1863–1867.

(RECEIVED 8/1/05; REVISION ACCEPTED 11/29/05;FINAL MATERIALS RECEIVED 2/1/06)

890 Volume 17—Number 10

Neural Preparation for Insight

at Copenhagen University Library on October 27, 2014pss.sagepub.comDownloaded from

Psychiatry Research: Neuroimaging 233 (2015) 278–284

Contents lists available at ScienceDirect

Psychiatry Research: Neuroimaging

http://d0925-49

n CorrE-m

journal homepage: www.elsevier.com/locate/psychresns

Neural correlates of response inhibition in children withattention-deficit/hyperactivity disorder: A controlled version of thestop-signal task

Tieme W.P. Janssen a,n, Dirk J. Heslenfeld a, Rosa van Mourik a, Gordon D. Logan b,Jaap Oosterlaan a

a VU University Amsterdam, Van der Boechorststraat 1, 1081 BT Amsterdam, Netherlandsb Vanderbilt University, 111 21st Street, Nashville, TN 37203, USA

a r t i c l e i n f o

Article history:Received 22 September 2014Received in revised form18 March 2015Accepted 7 July 2015Available online 9 July 2015

Keywords:Magnetic Resonance ImagingPsychopathologyPsychiatry

x.doi.org/10.1016/j.pscychresns.2015.07.00727/& 2015 Elsevier Ireland Ltd. All rights rese

esponding author.ail address: twp.janssen@vu.nl (T.W.P. Janssen

a b s t r a c t

The stop-signal task has been used extensively to investigate the neural correlates of inhibition deficits inchildren with ADHD. However, previous findings of atypical brain activation during the stop-signal taskin children with ADHD may be confounded with attentional processes, precluding strong conclusions onthe nature of these deficits. In addition, there are recent concerns on the construct validity of the SSRTmetric. The aim of this study was to control for confounding factors and improve the specificity of thestop-signal task to investigate inhibition mechanisms in children with ADHD. FMRI was used to measureinhibition related brain activation in 17 typically developing children (TD) and 21 children with ADHD,using a highly controlled version of the stop-signal task. Successful inhibition trials were contrasted withcontrol trials that were comparable in frequency, visual presentation and absence of motor response. Wefound reduced brain activation in children with ADHD in key inhibition areas, including the right inferiorfrontal gyrus/insula, and anterior cingulate/dorsal medial prefrontal cortex. Using a more stringentcontrolled design, this study replicated and specified previous findings of atypical brain activation inADHD during motor response inhibition.

& 2015 Elsevier Ireland Ltd. All rights reserved.

1. Introduction

Almost two decades ago, Barkley postulated an influentialmodel on impaired response inhibition as the underlying deficit inattention-deficit/hyperactivity disorder (ADHD) (Barkley, 1997).According to that model, impaired response inhibition leads todeficits in other executive function (EF) domains and the pheno-typic manifestation of ADHD. This model has led to an extendedliterature on EF in ADHD, with emphasis on inhibitory functioning.The stop task, which has been used extensively to investigateBarkley’s model, requires participants to withhold a motor re-sponse to a frequently presented go signal when prompted by aninfrequent and unpredictable stop signal (Logan and Cowan, 1984;Logan et al., 1984). The speed of the inhibition process appears tobe slower in childrenwith ADHD, as reflected in slower stop-signalreaction times (SSRT) (Oosterlaan et al., 1998).

However, two more recent meta-analyses on the stop task,utilizing an extended literature and including moderator variables,

rved.

).

question the interpretation of slower SSRT in children with ADHDas reflecting poor inhibition (Lijffijt et al., 2005; Alderson et al.,2007). Instead, the authors conclude that differences in SSRT maybe confounded by general slowing in mean reaction time (MRT)and increased reaction time variability (RTV), which is more in linewith a general deficit in attentional or cognitive processing.

Neuroimaging studies using the stop task in typically devel-oping (TD) participants showed that successful stopping activatesa brain network comprising the inferior frontal gyrus (IFG)/ante-rior insula, dorsal medial prefrontal cortex (dmPFC) including thepre-supplementary motor area (pre-SMA)/SMA and dorsal ante-rior cingulate cortex (ACC), and striatal and subthalamic nuclei(Swick et al., 2011). A recent meta-analysis (McCarthy et al., 2014)of five stop task studies in children with ADHD showed reducedactivation in bilateral IFG/Ins, right medial frontal gyrus, and rightsuperior and middle frontal gyri. Partially overlapping results werefound in another meta-analysis (Hart et al., 2013) of 15 studiesusing the stop task or go–nogo (GNG) tasks, with reduced activa-tion for ADHD in the right IFG/Ins, right SMA and ACC, right tha-lamus, left caudate and right occipital cortex. Contradicting resultsbetween the two meta-analyses may be explained by the inclusionof GNG task studies in Hart et al. (2013).

Table 1Group characteristics and task performance

ADHD TD Between-groupdifference(n ¼ 21) (n ¼ 17)

M SD M SD F(1,36) p

Demographic dataAge (years) 10.63 1.11 10.28 1.21 0.82 nsIQ 98.64 15.91 108.74 16.08 3.75 nsGender (M/F) 19/2 N/A 13/4 N/A 1.39a ns

DBDRS parentsInattention 21.24 3.63 3.24 2.51 300.33 o0.001Hyperactivity/ 19.00 7.38 3.11 2.25 73.09 o0.001

ImpulsivityDBDRS teacher

Inattention 14.95 5.53 1.48 1.83 92.34 o0.001Hyperactivity/ 13.38 4.97 2.37 3.11 63.37 o0.001Impulsivity

Stop TaskRuns (number) 7.57 0.60 7.88 0.33 3.67 nsCorrect Stop (%) 48.18 3.02 49.59 1.48 3.10 nsMRT (ms) 530.89 113.77 486.68 97.36 1.61 nsCV RT 0.25 0.05 0.21 0.06 3.12 nsSSD (ms) 235.07 92.68 249.94 87.08 0.26 nsSSRT (ms) 295.82 56.26 236.74 33.64 14.50 0.001CommissionErrors

10.29 6.69 7.88 5.96 1.34 ns

Omission Errors 7.10 6.88 3.29 3.82 4.14 0.049

Note. DBDRS¼Disruptive Behavior Disorders Rating Scale; MRT¼mean reactiontime on correct Go trials; CV¼coefficient of variation; SSD¼ stop-signal delay;SSRT¼stop signal reaction time.

a χ2(1)

T.W.P. Janssen et al. / Psychiatry Research: Neuroimaging 233 (2015) 278–284 279

Although there is convincing evidence for atypical brain acti-vation in ADHD during the stop task, the interpretation of thesefindings is challenging. One major methodological concern for thestop task is the confounding attentional capture effect of in-frequent stop stimuli (Sharp et al., 2010; Pauls et al., 2012), whichis not controlled with the conventional contrast between stop andgo conditions. Furthermore, several brain areas including the rIFG,which are activated during the stop task, are also activated inoddball paradigms and are part of a right lateralized ventral at-tentional system (Corbetta et al., 2002; Hampshire et al., 2010;Rubia et al., 2010c). These findings suggest that typical stop taskactivations may be confounded with attentional processes.

Particularly, the functional role of the rIFG is subject to debate,with some studies supporting a crucial role in detection of salientstimuli (Hampshire et al., 2010; Sharp et al., 2010), while otherstudies support a specific role in inhibition (Aron et al., 2004), andagain other studies supporting both functions (Verbruggen et al.,2010). This debate is particularly relevant for ADHD when con-sidering the possibility that slower SSRT in ADHD may be ex-plained by a deficit in attention (Lijffijt et al., 2005; Alderson et al.,2007) rather than an inhibition deficit. However, previous stoptask fMRI studies in ADHD have not controlled for attentionalcapture.

A few studies with the stop task have attempted to control forattentional capture in healthy adult populations with differentresults. Sharp et al. (2010) added infrequent continue signals to thestop task to control for attentional capture. Brain activation for thecontrol and successful inhibition conditions overlapped in therIFG, with only activation in the pre-SMA being uniquely asso-ciated with inhibition. Recent research however suggests thatcontinue signals may engage alternative strategies, which couldviolate stop task assumptions (Bissett and Logan, 2014). In con-trast, De Ruiter et al. (2012) found successful inhibition to be re-lated to activation in both IFG and pre-SMA after controlling forattentional capture using a different control method.

The current study aimed to improve our understanding of in-hibition deficits in children with ADHD by delineating inhibition-related brain activation during a stop task that controls for theattentional capture effect of stop stimuli. Based on previous stu-dies, we hypothesized that children with ADHD will show lessactivation in the dmPFC than TD children, and in the case of aspecific inhibitory role for the rIFG in children, will show reducedactivation in the rIFG as well. In accordance to Alderson et al.(2007) and Lijffijt et al. (2005), we expected that children withADHD will perform worse than TD children, with evidence forinhibition problems (increased SSRT), but also for more generalattentional problems (increased MRT, RTV, omission errors). Fi-nally, additional analyses were performed to assess error-relatedbrain activation during failed inhibition.

2. Methods

2.1. Participants

Thirty-eight right-handed children aged between 8 and 13years participated in this study (after final exclusion, see below),with 21 children in the ADHD group (19 males, 2 females), and 17children in the TD group (13 males, 4 females), see Table 1. In-clusion required an estimated full scale IQZ70 measured with ashort version of the Wechsler Intelligence Scale for Children(WISC-III; Wechsler, 1991), using the subtests Vocabulary, Ar-ithmetic, Block Design and Picture Arrangement. Children wereexcluded if there was a known history of neurological conditions,presence of brain anomalies as assessed by a neuroradiologist (2children with ADHD), or failure to meet basic task demands of at

least 5 runs with 470% correct go trials (1 child with ADHD).Parents and children aged 12 years or older signed informed-consent. The study was conducted according to the Declaration ofHelsinki, and approved by the ethics committee of the VU MedicalCentre (Amsterdam, The Netherlands).

The ADHD group was recruited through outpatient mentalhealth facilities in the Amsterdam area. All children obtained aclinical diagnosis of ADHD according to the DSM-IV (AmericanPsychiatric Association, 1994) as established by a child psychiatrist.ADHD diagnosis was confirmed with the parent version of theDiagnostic Interview Schedule for Children (DISC-IV; Shaffer et al.,2000), and by parent and teacher ratings on the Disruptive Be-havior Disorders Rating Scale (DBDRS; Pelham et al., 1992), whichrequired scores above the 90th percentile for parents and teachers.According to DISC criteria, 19 children fulfilled ADHD combinedsubtype criteria and 2 children met ADHD inattentive subtypecriteria. Exclusion criteria were comorbidity with other psychiatricdisorders, except oppositional defiant disorder (as assessed withthe DISC). Two children were medication naïve, and 19 childrendiscontinued stimulant medication at least 48 h before testing.

The TD group was recruited through local advertisement and inprimary schools in the Amsterdam area. TD children were requiredto obtain normal scores on parent and teacher reported DBDRS(o90th percentile) and to be free of any psychiatric disorder.

2.2. Stimuli and task

The stop task involved four trial types: go trials, stop trials andtwo types of trials that were used to control for confounding ac-tivation during successful and failed stop trials (see Fig. 1). The gotrials involved left or right pointing airplanes requiring a buttonpress with the left or right index finger, respectively. Each trialstarted with a white fixation cross, centred on a black backgroundfor 500 ms, followed by a 1500 ms go stimulus. Inter-trial-intervalsvaried randomly between 1000 ms and 5000 ms. In a randomly

Fig. 1. Trial types in controlled stop task; SI¼successful inhibition; FI¼failed inhibition. Note that in the SI-control condition, the go stimulus (airplane) is not followed by aresponse.

T.W.P. Janssen et al. / Psychiatry Research: Neuroimaging 233 (2015) 278–284280

selected 16.6% of the trials, go stimuli were followed by a visualstop signal (a white cross) superimposed on the go stimulus, re-quiring the participants to withhold their response. At the start ofthe experimental session, stop-signal delay (SSD) was set to theaverage SSD obtained in the preceding training session, whichtook place outside the scanner. For the training and experimentalsessions, the SSD between the go and stop stimuli was adaptedtrial-by-trial using an online tracking algorithm which increasedor decreased the delay by 50 ms, depending on whether or not theprevious stop trial resulted in successful inhibition (Logan et al.,1997). This procedure yielded approximately 50% successful in-hibitions (SI) and 50% failed inhibitions (FI). In control trials for SI(SI-C), which were randomly presented in 8.3% of the trials and asfrequently as SI trials (half of 16.6% stop trials), the stop signalappeared first and was followed after the current SSD by the gostimulus. This trial type was designed to be analogous to SI trials in(1) stimulus complexity, controlling for differences in visual pro-cessing, (2) frequency, controlling for attentional capture, and(3) the absence of motor response, to isolate neural activationspecifically related to active response inhibition (Heslenfeld andOosterlaan, 2003). In control trials for FI (FI-C), which were ran-domly presented in 8.3% of the trials and as frequent as FI trials,the stop signal appeared after a response had been made (incontrast to FI, where the stop signal preceded the response). Thedelay between the response and stop stimuli on FI-C trials variedconcordantly with SSD. This trial type controlled for the same is-sues as SI-C while allowing a motor response, to obtain a morespecific measure of error related neural processes (Heslenfeld andOosterlaan, 2003). Note that all events within each trial occurredwithin several hundred milliseconds, such that the resulting fMRIresponses will be sensitive to the processes initiated by the type oftrial as such, rather than the order of the individual events.

Participants first practiced two runs outside the scanner, andsubsequently, one practice run and eight experimental runs of 60trials were administered in 30 minutes with trials presented in apseudo-randomized order. Participants were instructed to respondboth quickly and accurately to the go stimuli, and to refrain fromresponding when prompted with a stop signal. They were told thatthey would be unable to withhold their responses on all stop trials,and that they should not wait for the stop stimulus. Furthermore,they were instructed that some trials started with the stop sti-mulus and that these instances did not require a response, andthat occasionally a stop sign followed their response on go trials.

2.3. fMRI image Acquisition

Images were acquired on a 1.5-T Siemens Sonata scanner(Siemens Medical Systems, Erlangen, Germany), equipped with avolume head coil. Stimuli were viewed through a mirror mountedon the head coil. Functional images were collected using a T2*-weighted echo-planar imaging (EPI) sequence scanning the wholebrain with 20 5.0-mm slices (TR¼2000 ms, TE¼60 ms, FA¼90°,64�64 matrix, 3.0�3.0 mm2 in-plane resolution, gap 20%). Eightfunctional runs (115 volumes each) were collected. A 3-D anato-mical scan was collected after the experimental session using a T1-weighted MP-Rage sequence (TR¼2700 ms, TE¼3.43 ms,TI¼1000 ms, FA¼7°, 256x160 matrix, 1.0�1.0 mm2 in-plane re-solution, 160 1.25 mm slices).

2.4. fMRI data analysis

MRI data were analysed with Brainvoyager QX-2.3 software(Maastricht, the Netherlands). Preprocessing steps involved: ex-clusion of the first two volumes of each run from the analysis toallow longitudinal magnetization to arrive at a steady state; rea-lignment of volumes to the third volume of each run with a rigid-body 3-D motion correction; slice scan time correction; 3-D spatialsmoothing with a 6-mm fullwidth at half maximum (FWHM)Gaussian kernel; high-pass filtering (0.02 Hz) to remove low fre-quencies; and low-pass filtering with a 3-s FWHM Gaussian kernelto remove high frequencies. Functional scans were coregistered toeach individual anatomical scan, spatially normalized to Talairachspace with the 9-parameter landmark method, and resampled at3�3�3 mm3 resolution (Goebel et al., 2006).

At the individual level, blood oxygen-level dependent (BOLD)responses of each voxel in each run were modelled with a generallinear model (GLM) including five experimental regressors andseven nuisance regressors. The first five regressors accounted forsuccessful inhibition (SI), successful inhibition control (SI-C), failedinhibition (FI), failed inhibition control (FI-C) and correct go trials.The last seven accounted for motion within each run with threetranslation and three rotation parameters in x, y, and z dimensions,and error trials that included erroneous responses other thanfailed inhibitions (i.e., commission and omission errors on go andcontrol trials), which were modelled as regressors of no interest.The hemodynamic response to each event was modelled by con-volving each regressor with a standard two-gamma HRF. Beta

Fig. 2. Statistical maps for the successful and failed inhibition contrasts for the TD and ADHD groups. Significant activation clusters with a cluster-size corrected threshold ofpo0.05 overlaid on the averaged Talairach-normalized image of all children. Note that the lower-bound in the TD group is t(16)¼2.12 for po0.05; R¼right hemisphere,L¼ left hemisphere; z¼vertical Talairach coordinate; x¼sagittal coordinate; t¼t-statistic.

T.W.P. Janssen et al. / Psychiatry Research: Neuroimaging 233 (2015) 278–284 281

estimates were obtained for each regressor by fitting the con-volved model to the voxel time series after correction for temporalautocorrelation (Goebel et al., 2006).

Eleven fMRI runs of 10 subjects were excluded due to failure tomeet criteria of 470% correct go trials (1 run for ADHD) or astrategic change of solely attending to the go instructions as re-flected in a SSD of 0 ms (2 runs for one child with ADHD), or due toexcessive movement during scanning with more than 3 mmtranslation in x, y, or z dimensions (6 runs for ADHD, 2 for TD).

At the group level, random effect second-level analyses wereperformed separately for the ADHD and TD groups, resulting instatistical parametric maps for SI versus SI-C (successful inhibitioncontrast), and FI versus FI-C (failed inhibition contrast). The un-corrected maps with a voxelwise threshold of po0.05 were cor-rected for multiple comparisons using cluster-size thresholding byMonte Carlo simulations to obtain significant clusters at po0.05 atthe whole brain level (required cluster threshold¼92 voxels). Theresulting activation clusters from the corrected TD and ADHDsingle-group maps were then analysed as inhibition-related re-gions of interest (ROI) for group differences. For the ROI analyses,individual subject ROI beta weights were calculated and extractedin Brainvoyager and used as dependent variables in groupcomparisons.

2.5. Statistical analysis

Demographic and performance data were compared betweengroups with ANOVA. Motion during the scanning session wastested for group differences with multivariate ANOVA. Assump-tions of the stop task such as independence of go and stop pro-cesses (MRT4mean signal-respond time), and comparability of RTskew and RT slowing were assessed (Verbruggen et al., 2013). RTslowing was tested with generalized estimated equations (GEE).

The extracted individual subject ROI beta weights were statis-tically analysed with GLMs in SPSS (Version 20). The alpha-levelwas Bonferroni-adjusted for the number of clusters within eachcontrast.

Variables for analysis of the performance data were number ofomission (no response to go stimulus) and commission (incorrectresponse) errors during go trials, mean reaction time (MRT) of

correct go trials, percentage correct go trials (number of go trials/total number of errors), mean SSD, reaction time variability(coefficient of variation [CV]: standard deviation/MRT), and SSRT,which was computed by subtracting SSD from MRT (Logan, 1994).Spearman correlations were performed between ROI betas, DBDRSscales and task parameters. Only ROI betas of activation clustersthat differed between groups were used for these analyses. Alphawas set at 0.05, two-tailed.

3. Results

3.1. Group characteristics and behavioural data

Table 1 summarizes the demographic and task performancedata. Groups did not differ on age. There was a non-significanttrend (p¼0.061) for higher IQ in the TD group compared to theADHD group. However, IQ did not correlate significantly with anyof the outcome measures in this study (p-values40.162). Mean goRT was slower than mean signal-respond RT, F(1,36)¼52.27,po0.001, no differences were found between groups in skewnessof go RT distributions, F(1,36)¼0.32, p¼0.58, RT slowing, Wald χ2(1)¼2.07, p¼0.15, or percentage of successful inhibition, indicat-ing the assumptions of the race model were met. Furthermore,groups did not differ significantly on number of runs on the stoptask. At last, the ADHD group showed slower SSRTs and mademore omission errors than the TD group.

Multivariate ANOVA showed no significant differences betweengroups on translation or rotation parameter in x, y, or z dimen-sions, F(6,31)¼1.05, pr0.50.

3.2. Brain activations during successful and failed inhibition

Brain activation for the successful inhibition and failed inhibi-tion contrasts, for both groups separately, are shown in Fig. 2 andTable 2.

3.2.1. Successful inhibition contrastThe successful inhibition contrast comparing SI and SI-C trials

showed increased activation during SI trials in the TD group forthe right IFG/insula, left insula, bilateral anterior cingulate cortex

Table 2Activated brain regions for successful and failed inhibition contrasts, separately for TD and ADHD, and comparisons between groups

Areaa Side Peak voxel Brodmann area Voxels Between-group differenceb

x y z n p F(1,36) p η2 Direction

Successful Inhibition ContrastTD

Inferior frontal cortex, insula, claustrum R 47 16 3 47,44,45,13 365 o0.001 9.03* 0.005 0.20 TD4ADHDInsula, claustrum L �34 10 3 13 107 0.016 0.33 nsAnterior cingulate/anterior medial frontal cortex L/R 5 34 15 32,24,10 100 0.028 8.34* 0.007 0.19 TD4ADHDMedial/superior frontal cortex R 20 55 27 9,8,6 92 0.050 6.86† 0.013 0.16 TD4ADHD

ADHDInsula, middle/superior frontal cortex, claustrum R 32 40 30 13,9.10 330 o0.001 1.16 nsInsula, claustrum L �31 16 6 13 130 0.005 3.46 nsAnterior cingulate L/R 5 19 33 32,24,9 117 0.010 2.05 ns

Failed inhibition contrastADHD

Precentral gyrus, posterior insula R 38 �11 9 6,4,13 218 o0.001 8.34* 0.007 0.19 ADHD4TDAnterior cingulate/dorsal medial frontal cortex L/R 5 13 38 32,24 175 o0.001 1.95 nsPrecentral gyrus, postcentral gyrus L �43 �17 33 6,4,3 115 0.007 7.48* 0.010 0.17 ADHD4TDMiddle/superior frontal cortex R 29 40 33 9,10 92 0.050 4.70 ns

a Significant activation clusters with a clustersize corrected threshold of po0.05; R¼right hemisphere, L¼ left hemisphere; peak voxel is the most significant voxel inTalairach space; n¼number of voxels

b Univariate GLMs to test for group differences for each region of interest (ROI).* Significant after Bonferroni-correction.† near significant at a Bonferroni-adjusted alpha level of p¼0.0125, with p¼0.0128.

T.W.P. Janssen et al. / Psychiatry Research: Neuroimaging 233 (2015) 278–284282

(ACC)/anterior medial frontal cortex and right medial/superiorfrontal cortex. In the ADHD group this contrast showed increasedactivation during SI trials in the right insula and middle/superiorfrontal cortex, left insula and ACC.

3.2.2. Failed inhibition contrastThe failed inhibition contrast comparing FI and FI-C trials

showed no activation in the TD group. In the ADHD group thiscontrast showed activation in bilateral premotor/primary motorareas, the ACC/dorsal medial frontal cortex and right middle/su-perior frontal cortex.

3.3. Between-group comparisons

The within-group activation clusters were tested as inhibition-related regions of interest for group differences. p-Values wereBonferroni-adjusted for the successful inhibition contrast (for TDclusters p¼0.05/4¼0.0125, for ADHD clusters p¼0.05/3¼0.0167)and failed inhibition contrast (for the ADHD clusters p¼0.05/4¼0.0125), based on the number of clusters per group. Results ofthe between-group analyses are shown in Table 2.

3.3.1. Successful inhibition contrastFor the ROIs based on the TD group, the ADHD group activated

the right IFG/insula and ACC/anterior medial frontal cortex to alesser extend than the TD group. There was also a near-significanttrend for reduced activation in the ADHD group in a more dorsallylocated dmPFC area including the right pre-SMA, extending to theright superior frontal cortex. No differences between groups werefound for the ROIs based on the ADHD group.

3.3.2. Failed inhibition contrastFor the ROIs based on the ADHD group, the TD group showed

less activation in bilateral premotor/primary motor areas com-pared to the ADHD group.

3.3.3. Sensitivity analysisAnalyses were repeated with only male subjects, showing the

same pattern of results.

3.4. Correlations

A strong correlation was obtained between parent reportedhyperactivity/impulsivity symptoms in the ADHD group and acti-vation in the right motor cortex during FI, r(19)¼0.66, p¼0.001.

4. Discussion

The present study aimed to advance the understanding of in-hibition deficits in children with ADHD, by isolating inhibition-related brain activation in a highly controlled stop task. In contrastto previous studies using the stop task, our task controls for theconfounding effects of attentional capture, visual presentationdifferences, and motor responses. As hypothesized, children withADHD had a slower inhibition process (increased SSRT) and mademore omission errors. No evidence was found for increased MRTand RTV. Both the TD and ADHD groups activated bilateral IFG/insular regions and the ACC. As expected, children with ADHDactivated the rIFG/insula and dmPFC less than TD children duringsuccessful inhibition.

The imaging results of this study are largely in line with themeta-analysis of McCarthy et al. (2014) and Hart et al. (2013),showing reduced activation in rIFG/insula and dmPFC areas. TherIFG is part of a putative inhibition network, connected via a directpathway with the subthalamic nucleus (STN), both of which areconnected with the pre-SMA (Aron, 2007). Aron et al. (2007)propose that the rIFG implements inhibition at a neural level byactivating the STN, which activates the globus pallidus, resulting inthalamo-motorcortical inhibition. The pre-SMA could have a con-flict monitoring function or implement neural inhibition directlyvia the STN. Our results showed reduced activation in key areas ofthis inhibition network in ADHD, including the rIFG/insula, ACCand pre-SMA. The current findings support an inhibition-relateddysfunction in children with ADHD.

Performance data in our study are also consistent with the idea

T.W.P. Janssen et al. / Psychiatry Research: Neuroimaging 233 (2015) 278–284 283

of an inhibition-dysfunction in ADHD. SSRT differed significantlybetween groups, with the ADHD group having slower SSRTs thanthe TD group. Both SSD and MRT did not significantly differentiategroups. However, it cannot be ruled out that the SSRT differencebetween groups is partially driven by a difference in MRT as wassuggested by Alderson et al. (2007).

Despite clear evidence for atypical brain activation in childrenwith ADHD during the stop task, previous neuroimaging studieseither showed no performance differences between subjects withADHD and controls (Pliszka et al., 2006; Rubia et al., 2010a; Cubilloet al., 2014), or found evidence for attention related problems inADHD in increased RTV (Rubia et al., 2005, 2008, 2011), or higherrates of omission errors (Rubia et al., 2005). Only two studiesfound lower rates of probability of inhibition (Rubia et al., 1999;Rubia, 2001), but no SSRT differences. These behavioural findingschallenge the interpretation of the accompanying neuroimagingresults.

Another important observation in this study is the role ofanterior insular cortex during successful inhibition. Swick et al.(2011) emphasized in their meta-analysis the unexpected greaterprominence of insular activation compared to the IFG activationduring GNG and stop tasks. They suggested two possible ex-planations. First, some studies interpret activation foci in the in-sula as IFG activations due to localisation error, or second, spatialsmoothing methods can blur spatially distinct patterns and lead toerrors in localisation of brain activation. The insula is described asa highly integrative area, and is found in a wide range of cognitivetasks (Kurth et al., 2010). Singer et al. (2009) proposed that theinsula plays an important role in the signaling of uncertainty. Al-though our stop task controlled for attentional capture, SI trialsand SI-C trials were somewhat different in respect to uncertainty.SI trials started with a go signal, for which the child was uncertainwhether or not a stop-signal would occur afterwards. In contrast,SI-C were as frequently and randomly presented as SI trials, butstarted with a stop signal, with no uncertainty about immediatesubsequent events. In conclusion, the bilateral insular activation inour study might also be related to uncertainty, as well asinhibition.

Error related brain activation in the stop task was not as ex-pected. For the TD group no activity was found, whereas for theADHD group activation was found in a typical error/conflictmonitoring area, the ACC (Shenhav et al., 2013), and bilateralmotor areas. The absence of activation in the TD group could bedue to a differential behavioural response to the control conditionas compared to the ADHD group. The control condition was equalto a normal go trial; however, the manual response was followedby a stop-signal to control for confounding visual input. Possibly,TD children interpreted the appearance of the stop-signal, despitethe task instructions, as an error, removing (or reducing) the re-sulting activation from the failed inhibition condition. The ADHDgroup also activated right motor areas during FI, which correlatedwith hyperactivity/impulsivity as reported by parents. Possibly,this finding might be explained by a lower frustration threshold inADHD (Mick et al., 2005), accompanied by more motor rest-lessness in response to errors in ADHD.

Some limitations of this study should be noted. First, mostchildren in the ADHD group were on stimulant medication. Al-though medication use was discontinued before testing, long-termeffects of chronic treatment with MPH on brain functioning havebeen reported, with one study showing normalization (Bush et al.,2008), while others found MPH to be insufficient to normalizeneurofunctional deficits (Schweitzer et al., 2004; Konrad et al.,2007). For the stop task in particular, activation differences be-tween children with ADHD and TD in the right superior frontalgyrus were larger in treatment naive children in the meta-study ofMcCarthy et al. (2014). In the current study, we found a near

significant effect for reduced activation in the right medial/su-perior frontal gyrus in ADHD during successful inhibition, whichdid not survive Bonferroni-correction. It cannot be ruled out thatchronic stimulant use diminished group differences in this brainarea. Furthermore, acute medication withdrawal effects may haveaffected our results. The meta-study by McCarthy et al. (2014)found an effect of medication washout-length on brain activation.More specifically, shorter washout periods meant fewer activationdifferences compared to controls in the left medial frontal gyrus,and longer washout periods meant more activation differencescompared to controls in the right precuneus. These results suggestthat acute effects of treatment cessation, similarly to long-termmedication effects, are associated with normalized brain activity.Consequently, brain activation differences in our study may havebeen reduced. However, our finding of reduced activation in rIFG isin line with fMRI studies in medication-naïve boys with ADHD(Rubia et al., 2005, 2010b), increasing the confidence in ourfindings.

Another limitation is that the control conditions may have in-duced inhibition-related activation that in turn would have di-minished differences obtained in our successful and failed in-hibition contrasts. Although task instructions clearly stated thatcontrol trials did not require a response, their infrequency com-pared to go trials could have triggered partial inhibition, compar-able to a nogo trial in a GNG task. The successful inhibition con-trast showed activation in key motor inhibition areas, but effects inother brain areas, especially basal ganglia nuclei such as thestriatum (Zandbelt and Vink, 2010; Vink et al., 2014), could havebeen diminished; although this could also be the result of thecluster-size thresholding method, which may be less likely toshow smaller activation areas. Future study designs of the stoptask could use a neutral stimulus in the control condition to reducethe go/no-go inhibition effect. A second limitation may be thatonly SI trials involve a fast redirection of attention from go to stopinstructions (cognitive set-shifting), in contrast to control trials(SI-C). This may have contributed to differences between SI and SI-C trials even though both types of trials were equated for stimu-lus-related, response-related, and probability-related processes(i.e. attentional capture).

In conclusion, this study confirmed hypoactivation in key in-hibition areas in children with ADHD, while controlling for theconfounding effects of attentional capture, visual presentationdifferences, and motor response. Furthermore, these findings werecomplemented by evidence for inhibitory control problems at thebehavioural level. To our knowledge, this is the first study inchildren with ADHD that incorporates stringent control conditionsin the stop task in order to isolate inhibition-related brainactivation.

References

Alderson, R., Rapport, M., Kofler, M., 2007. Attention-deficit/hyperactivity disorderand behavioral inhibition: a meta-analytic review of the stop-signal paradigm.J. Abnorm. Child Psychol. 35, 745–758. http://dx.doi.org/10.1007/s10802-007-9131-6.

American Psychiatric Association. (1994). Diagnostic and statistical manual ofmental disorders (4th ed.). Washington, DC.

Aron, A., 2007. The neural basis of inhibition in cognitive control. Neuroscientist 13,214–228. http://dx.doi.org/10.1177/1073858407299288.

Aron, A., Behrens, T., Smith, S., Frank, M., Poldrack, R., 2007. Triangulating a cog-nitive control network using diffusion-weighted magnetic resonance imaging(MRI) and functional MRI. J. Neurosci. 27, 3743–3752. http://dx.doi.org/10.1523/JNEUROSCI.0519-07.2007.

Aron, A., Robbins, T., Poldrack, R., 2004. Inhibition and the right inferior frontalcortex. Trends Cogn. Sci. 8, 170–177. http://dx.doi.org/10.1016/j.tics.2004.02.010.

Barkley, R.A., 1997. Behavioral inhibition, sustained attention, and executive func-tions: constructing a unifying theory of ADHD. Psychol. Bull. 121, 65–94.

Bissett, P.G., Logan, G.D., 2014. Selective stopping? Maybe not. J. Exp. Psychol.: Gen.

T.W.P. Janssen et al. / Psychiatry Research: Neuroimaging 233 (2015) 278–284284

143, 455–472. http://dx.doi.org/10.1037/a0032122.Bush, G., Spencer, T.J., Holmes, J., Shin, L.M., Valera, E.M., Seidman, L.J., Makris, N.,

Surman, C., Aleardi, M., Mick, E., Biederman, J., 2008. Functional magnetic re-sonance imaging of methylphenidate and placebo in attention-deficit/hyper-activity disorder during the multi-source interference task. Arch. Gen. Psy-chiatry 65, 102–114. http://dx.doi.org/10.1001/archgenpsychiatry.2007.16.

Corbetta, M., Kincade, J.M., Shulman, G.L., 2002. Neural systems for visual orientingand their relationships to spatial working memory. J. Cogn. Neurosci. 14,508–523. http://dx.doi.org/10.1162/089892902317362029.

Cubillo, A., Smith, A., Barrett, N., Giampietro, V., Brammer, M., Simmons, A., Rubia,K., 2014. Shared and drug-specific effects of atomoxetine and methylphenidateon inhibitory brain dysfunction in medication-naive ADHD boys. Cereb. Cortex24, 174–185. http://dx.doi.org/10.1093/cercor/bhs296.

De Ruiter, M.B., Oosterlaan, J., Veltman, D.J., van den Brink, W., Goudriaan, A.E.,2012. Similar hyporesponsiveness of the dorsomedial prefrontal cortex inproblem gamblers and heavy smokers during an inhibitory control task. DrugAlcohol Depend. 121, 81–89. http://dx.doi.org/10.1016/j.drugalcdep.2011.08.010.

Goebel, R., Esposito, F., Formisano, E., 2006. Analysis of functional image analysiscontest (FIAC) data with brainvoyager QX: from single-subject to corticallyaligned group general linear model analysis and self-organizing group in-dependent component analysis. Hum. Brain Mapp. 27, 392–401. http://dx.doi.org/10.1002/hbm.20249.

Hampshire, A., Chamberlain, S.R., Monti, M.M., Duncan, J., Owen, A.M., 2010. Therole of the right inferior frontal gyrus: inhibition and attentional control.NeuroImage 50, 1313–1319. http://dx.doi.org/10.1016/j.neuroimage.2009.12.109.

Hart, H., Radua, J., Nakao, T., Mataix-Cols, D., Rubia, K., 2013. Meta-analysis offunctional magnetic resonance imaging studies of inhibition and attention inattention-deficit/hyperactivity disorder: exploring task-specific, stimulantmedication, and age effects. JAMA Psychiatry 70, 185–198. http://dx.doi.org/10.1001/jamapsychiatry.2013.277.

Heslenfeld, D.J., Oosterlaan, J., 2003. Abstracts of the Dutch Psychophysiology So-ciety Conference. J. Psychophysiol. 17, 137–141. http://dx.doi.org/10.1027//0269-8803.17.3.137.

Konrad, K., Neufang, S., Fink, G.R., Herpertz-Dahlmann, B., 2007. Long-term effectsof methylphenidate on neural networks associated with executive attention inchildren with ADHD: results from a longitudinal functional MRI study. J. Am.Acad. Child Adolesc. Psychiatry 46, 1633–1641. http://dx.doi.org/10.1097/chi.0b013e318157cb3b.

Kurth, F., Zilles, K., Fox, P.T., Laird, A.R., Eickhoff, S.B., 2010. A link between thesystems: functional differentiation and integration within the human insularevealed by meta-analysis. Brain Struct. Funct. 214, 519–534. http://dx.doi.org/10.1007/s00429-010-0255-z.

Lijffijt, M., Kenemans, J.L., Verbaten, M.N., van Engeland, H., 2005. A meta-analyticreview of stopping performance in attention-deficit/hyperactivity disorder:deficient inhibitory motor control? Journal of Abnormal Psychology 114,216–222. http://dx.doi.org/10.1037/0021-843X.114.2.216.

Logan, G., 1994. On the Ability to Inhibit Thought and Action: A Users’ Guide to theStop Signal Paradigm. Academic Press, San Diego.

Logan, G., Cowan, W., 1984. On the ability to inhibit simple and choice reaction timeresponses: a theory of an act of control. Psychol. Rev. 91, 295–327.

Logan, G., Cowan, W., Davis, K., 1984. On the ability to inhibit simple and choicereaction time responses: a model and a method. J. Exp. Psychol. Hum. Percep.Perform. 10, 276–291.

Logan, G., Schachar, R., Tannock, R., 1997. Impulsivity and inhibitory control. Psy-chol. Sci. 8, 60–64.

McCarthy, H., Skokauskas, N., Frodl, T., 2014. Identifying a consistent pattern ofneural function in attention deficit hyperactivity disorder: a meta-analysis.Psychol. Med. 44, 869–880. http://dx.doi.org/10.1017/S0033291713001037.

Mick, E., Spencer, T., Wozniak, J., Biederman, J., 2005. Heterogeneity of irritability inattention-deficit/hyperactivity disorder subjects with and without mood dis-orders. Biol.Psychiatry 58, 576–582. http://dx.doi.org/10.1016/j.biopsych.2005.05.037.

Oosterlaan, J., Logan, G.D., Sergeant, J.A., 1998. Response inhibition in AD/HD, CD,comorbid AD/HDþCD, anxious, and control children: a meta-analysis of studieswith the stop task. J. Child Psychol. Psychiatry 39, 411–425.

Pauls, A., O’Daly, O., Rubia, K., Riedel, W., Williams, S., Mehta, M., 2012. Methyl-phenidate effects on prefrontal functioning during attentional-capture andresponse inhibition. Biol. Psychiatry 72, 142–149. http://dx.doi.org/10.1016/j.biopsych.2012.03.028.

Pelham, W.E., Gnagy, E.M., Greenslade, K., Milich, R., 1992. Teacher ratings of DSM-III-R symptoms for the disruptive behaviour disorder. J. Am. Acade. ChildAdolesc. Psychiatry 31, 210–218.

Pliszka, S.R., Glahn, D.C., Semrud-Clikeman, M., Franklin, C., Perez, R., Xiong, J.,Liotti, M., 2006. Neuroimaging of inhibitory control areas in children with

attention deficit hyperactivity disorder who were treatment naive or in long-term treatment. Am. J. Psychiatry 163, 1052–1060. http://dx.doi.org/10.1176/appi.ajp.163.6.1052.

Rubia, K., 2001. Neuropsychological analyses of impulsiveness in childhood hy-peractivity. Br. J. Psychiatry 179, 138–143. http://dx.doi.org/10.1192/bjp.179.2.138.

Rubia, K., Cubillo, A., Smith, A., Woolley, J., Heyman, I., Brammer, M., 2010a. Dis-order-specific dysfunction in right inferior prefrontal cortex during two in-hibition tasks in boys with attention-deficit hyperactivity disorder compared toboys with obsessive-compulsive disorder. Hum. Brain Mapp. 31, 287–299. http://dx.doi.org/10.1002/hbm.20864.

Rubia, K., Cubillo, A., Smith, A.B., Woolley, J., Heyman, I., Brammer, M.J., 2010b.Disorder-specific dysfunction in right inferior prefrontal cortex during twoinhibition tasks in boys with attention-deficit hyperactivity disorder comparedto boys with obsessive-compulsive disorder. Hum. Brain Mapp. 31, 287–299.http://dx.doi.org/10.1002/hbm.20864.

Rubia, K., Halari, R., Mohammad, A., Taylor, E., Brammer, M., 2011. Methylphenidatenormalizes frontocingulate underactivation during error processing in atten-tion-deficit/hyperactivity disorder. Biol. Psychiatry 70, 255–262. http://dx.doi.org/10.1016/j.biopsych.2011.04.018.

Rubia, K., Halari, R., Smith, A.B., Mohammed, M., Scott, S., Giampietro, V., Taylor, E.,Brammer, M.J., 2008. Dissociated functional brain abnormalities of inhibition inboys with pure conduct disorder and in boys with pure attention deficit hy-peractivity disorder. Am. J. Psychiatry 165, 889–897. http://dx.doi.org/10.1176/appi.ajp.2008.07071084.

Rubia, K., Hyde, Z., Halari, R., Giampietro, V., Smith, A., 2010c. Effects of age and sexon developmental neural networks of visual–spatial attention allocation.NeuroImage 51, 817–827. http://dx.doi.org/10.1016/j.neuroimage.2010.02.058.

Rubia, K., Overmeyer, S., Taylor, E., Brammer, M., Williams, S.C., Simmons, A.,Bullmore, E.T., 1999. Hypofrontality in attention deficit hyperactivity disorderduring higher-order motor control: a study with functional MRI. Am. J. Psy-chiatry 156, 891–896.

Rubia, K., Smith, A., Brammer, M., Toone, B., Taylor, E., 2005. Abnormal brain acti-vation during inhibition and error detection in medication-naive adolescentswith ADHD. Am. J. Psychiatry 162, 1067–1075. http://dx.doi.org/10.1176/appi.ajp.162.6.1067.

Schweitzer, J.B., Lee, D.O., Hanford, R.B., Zink, C.F., Ely, T.D., Tagamets, M.A., Hoff-man, J.M., Grafton, S.T., Kilts, C.D., 2004. Effect of methylphenidate on executivefunctioning in adults with attention-deficit/hyperactivity disorder: normal-ization of behavior but not related brain activity. Biol. Psychiatry 56, 597–606.http://dx.doi.org/10.1016/j.biopsych.2004.07.011.

Shaffer, D., Fisher, P., Lucas, C.P., Dulcan, M.K., Schwab-Stone, M.E., 2000. NIMHDiagnostic Interview Schedule for Children Version IV (NIMH DISC-IV): de-scription, differences from previous versions, and reliability of some commondiagnoses. J. Am. Acad. Child Adolesc. Psychiatry 39, 28–38. http://dx.doi.org/10.1097/00004583-200001000-00014.

Sharp, D., Bonnelle, V., De Boissezon, X., Beckmann, C., James, S., Patel, M., Mehta,M., 2010. Distinct frontal systems for response inhibition, attentional capture,and error processing. Proc. Natl. Acad. Sci. U.S.A. 107, 6106–6111. http://dx.doi.org/10.1073/pnas.1000175107.

Shenhav, A., Botvinick, M.M., Cohen, J.D., 2013. The expected value of control: anintegrative theory of anterior cingulate cortex function. Neuron 79, 217–240.http://dx.doi.org/10.1016/j.neuron.2013.07.007.

Singer, T., Critchley, H.D., Preuschoff, K., 2009. A common role of insula in feelings,empathy and uncertainty. Trends Cogn. Sci. 13, 334–340. http://dx.doi.org/10.1016/j.tics.2009.05.001.

Swick, D., Ashley, V., Turken, U., 2011. Are the neural correlates of stopping and notgoing identical? Quantitative meta-analysis of two response inhibition tasks.NeuroImage 56, 1655–1665. http://dx.doi.org/10.1016/j.neuroimage.2011.02.070.

Verbruggen, F., Aron, A., Stevens, M., Chambers, C., 2010. Theta burst stimulationdissociates attention and action updating in human inferior frontal cortex. Proc.Natl. Acad. Sci. U.S.A. 107, 13966–13971. http://dx.doi.org/10.1073/pnas.1001957107.

Verbruggen, F., Chambers, C.D., Logan, G.D., 2013. Fictitious inhibitory differences:how skewness and slowing distort the estimation of stopping latencies. Psy-chol. Sci. 24, 352–362. http://dx.doi.org/10.1177/0956797612457390.

Vink, M., Zandbelt, B.B., Gladwin, T., Hillegers, M., Hoogendam, J.M., van denWildenberg, W.P.M., Du Plessis, S., Kahn, R.S., 2014. Frontostriatal activity andconnectivity increase during proactive inhibition across adolescence and earlyadulthood. Hum. Brain Mapp. 35, 4415–4427. http://dx.doi.org/10.1002/hbm.22483.

Wechsler, D., 1991. Weschler Intelligence Scale for Children, third edition. Psy-chological Corporation, San Antonio, TXC.

Zandbelt, B.B., Vink, M., 2010. On the role of the striatum in response inhibition.PLoS ONE 5. http://dx.doi.org/10.1371/journal.pone.0013848.