Ericsson verbal protocol

7/22/2019 Ericsson verbal protocol

1/67


2/67

PROTOCOL ANALYSISVERBAL REPORTS AS DATAREVISED EDITION

K. Anders EricssonandHerbert A. Simon

A Bradford BookThe MIT PressCambridge, MassachusettsLondon, England


3/67

@ 1984, 1993 Massachusetts Institute of TechnologyAll rights reserved. No part of this book may be reproduced in any form by any electronicLireZlu"i.A means (including photocopying, recording, or information storage andretrieval) without permission in writing fom the publisher'Printed and bound in the United States of America.Library of Congress Cataloging-in'Publication DataEricsson, K. Arders (Karl Anders)' 1947-- p.toot analysis :ebal trpors ar data / K. Anders Ericsson and Hebert A. Simon.

IN

- Rev. ed.P. cm."A Bradford Book."Includes bibliographical references and index.rsBN 0-262-0 iO+1 -t.- ISBN 0-262-s5023'7 (pbk')1. Verbal behavior-Reseach. 2. cognition-Research. 3. BehaviofismlPsvcholoev)-Research. 4. Psychology, Experimental'i. Siron,liruert Alexander, 191 II. Title.8F455.E68 L9931531c201098765

92-33817cIP


4/67

in any form by any electronicinformation storage and

and Herbert A. Simon.

3.Behaviorism

To

INGRID and KARL-OLOV ERICSSONand

to the mernory ofBDNA and ARTHUR SIMON

who taught us not to be afridto voice our thoughts

92-338L7CIP


5/67

TINTRODUCTION AND SUMMARY

After a long period of time during which stimulus-response relations wereat the focus ofattention, research in psychology is now seeking to under-stand in detail the mechanisms and internal structure of cognitiveprocesses that produce these relations. In the. limiting case, lve wouldlike to have process models so explicit that they could actually producethe predicted behavior from the information in the stimulus.This concern for the course of the cognitive processes has revivedinterest in finding ways to increase the temporal density of observationsso as to reveal intermediate stages of the processes. Increasingly, inves-tigators record the directions of the subject's gaze (eye movements), andthe intermediate behaviors (movements'or physical manipulations ofstimulus material) that precede the solution or criterion performance.Since data on intermediate processing are costly to gather and analyze, itis important to consider carefully how such data can be interpretedvalidly, and what contribution they can make to our understanding of thephenomena under study.One means frequently used to gain information about the course ofthe cognitive processes is to probe the subjects'internal states by verbalmethods. These methods are the topic of this monograph.USING YERBAL REPORTS: SOME ISSUES

There are several issues that we must deal with if we are to use subjects'reports as fundamental data in psychological experiments. First, we mustrespond to the strong doubts that have been expressed by manypsychologists in the past about the suitability of subjects' verbalizations asscientifc data. Second, we must consider the processing that must take


6/67

2 Protocol AnalYsisplace in ordef to transform subjects'behaviors (whether verbal orinto data. Third, $'e must examine how the encoding of behaviordata can be made objective and univocal, so that the resulting data will..hard" and not ,,soft." Fourth, we must be explicit about the tpresuppositions that are necessarily embedded in the encoding process,Finally, we must specify the processes that allow us to go backward fromthe data to the behavior and thence to inferences about the subjec6'thought processes.We offer a few comments on each of these five issues' They willreappear frequently as recurrent themes throughout the monograph.Doubts About Verbal Data

E$rccting Data from Be.. - rrrot verbal ferne lo.131"t. i" benal truu";';,tnns are_ ' - "o"tttto*"'---Whlcn-v-'-*"oorts is m

ing vo'']l^^ nnt concegators,neol* *tt *"tttheq:'gr:;::"" is tiuteretttl' -'i.i rtata'analsttnt:l::i:^^; ese mtlt :::i*; use oftions rr-', "*e .o mak' I'v -- rcehrrto cognitive Prtl;'; ririt Ptace' n*is*[t*:tlii'rt i**ten the subi#-'l*l""ii:Jot - rot*t ofiittPtttiue responsttP*,t of trained ob"introspectton'

Soft versus lfaril DSome investigatorsIi *nu"tt to simpt.nntt, which ar;;tln science'r'iUt" Uttt""tn databservation; theor;; ;bt"*ation-bittff. agreementhaviot'Even Psycho

Since the triumph of behaviorism over "introspectively" oriented compet'ing viewpoints, verbal reports have been suspect as data. More precisely,bJhaviorism and allied schools of thought have been schizophrenic aboutthe status of verbalizations as data. On the one hand, verbal responses(or key punches that are psychologically indistinguishable from verbal,"rponr.r, eicept that they are made with the finger instead of themouth) provide the basic data in standard experimental paradigms' In aconcept attainment experiment, the subjects say (or signal) 'yes" or "no"when a possible instance is presented to them. In a problem solving ex'periment, they report the answer when they find it' In a rote verbaliearning experiment, they say "DAX" when the stimulus syllable '.EF'is presented. The actual performance measures commonly uselatenciesan numbers of items correc' are derived from these responses' and theformer depend for their validity on the veridicality of the latter.on the other hand, modern psychology has been dubious about ver'balizations produced by subjects along the route to their solutions or ftnalresponses. Even more dubious has been the status of responses to ex'perimenter probes or retrospective answers to questions about prior be-havior. All of these sorts of verbal behavior are frequently dismissed asvariants of the discredited process of introspection (Nisbett & Wilson'tg77). Introspection, it has generally been argued' may be useful for thediscovery of psychological processes; it is worthless for verification. AsLashley Qgz3, p.352i said, in a vigorous and widely cited attack on themethod, ,.introspection may make the preliminary survey, but it must befollowed by the chain and transit of objective measutement'"


7/67

behaviors (whether verbal or not)ow the encoding of behavior intoso that the resulting data will bebe explicit about the theoreticalin the encoding process.that allow us to go backward fromto inferences about the subjects'of these five issues. They willthroughout the monograph.

"introspectively" oriented compet-suspect as data. More precisely,have been schizophrenic aboutthe one hand, verbal responsesindistinguishable from verbalwith the finger instead of theexperimental paradigms. In asay (or signal) ' es" or .,no"them, In a problem solving ex-they find it. In a rote verbalwhen the stimulus syllable ,.CEF'commonly uselatenciesfrom these responses, and theveridicality of the latter.has been dubious about ver-route to their solutions or finalthe status of responses to ex-to questions about prior be-are frequently dismissed asintrospection (Nisbett & Wilson,argued, may be useful for theis worthless for verification. Asand widely cited attack on thesurvey, but it must bemeasurement."

Introduction and Summary 3

Extracting Data from BehaviorThe notion that verbal reports provide possibly interesting but only infor-mal information, to be verifed by other data, has affected the ways inwhich verbalizations are collected and analyzed. If the purpose of obtain-ing verbal reports is mainly to generate hypotheses and ideas, inves-tigators need not concern themselves (and generally have not concernedthemselves) with methodological questions about data collection. As aresult, there is little published literature on such issues, the data-gathering and data-analysis methods actually used vary tremendously, andthe details of these methods are reported sketchily in research publica-tions that make use of such data.If we are to make rapid and continuing progress in understandinghuman cognitive processes, this state of,affairs is wholly unsatisfactory.In the first place, no clear guidelines are provided to distinguish il-legitimate "introspection" from the many forms of verbal output that areroutinely treated as datas passing the chain and transit test (see the ex-amples above). On what theoretical or practical grounds do we distin-guish between the subject's "yes" or "no" in a concept attainment experi-ment and his assertion that the hypothesis he is entertaining is "smallyellow circle"? In the second place, no distinctions are made among suchdiverse forms of verbalization as thinking-aloud (TA) protocols,retrospective responses to specific probes, and the classical introspectivereports of trained observers. All are jointly and loosely condemned as"introspection."Soft versus Harrl Data

Some investigators call verbal reports and verbal descriptions "soft data"in contrast to simple behavioral measures like latency or correctness ofresponse, which are referred to as "hard." What does this distinctionmean? In science one would like to maintain as clear a separation as pos-sible between data and theory. Data are supposed to derive directly fromobservation; theories are supposed to account for, explain, and predictthese observation-based data. Data are "hard" when there is intersub-jective agreement that they correspond to the facts of the observed be-havior.Even psychoanalytically or existentially oriented psychologists will


8/67

4 Protocol AnalYsisaccept response latencies as dateven though being possibly irrelevant'data for eiplaining behavior. When, however, an analyst codes a fve-'second description of a dream as "oral fixation," many psychologistswould argue that this encoding is not a datum but a subjective interpreta'tion of ti" dut. (i.e., of the verbal description of the dream). Surely,theory-laden inferences were required to derive the encoding from theverbai protocol. Data are regarded as "soft" to the degree that they in',orpor"i, such inferences, especially when the theoretical premises andruls of inference are themselves not completely explcit and objective'The problem with "soft" data is that different interpreters making dif-ferenl inferences will not agree in their encodings, and each interpfeter islikely, wittingly or not, to arrive at an interpretation that is favorable tohis theoretical orientation.The hard-soft distinction is orthogonal to the distinction betweenverbal and non-verbal. The same problems of inference can emerge inobservers' attempts to understand non-verbal events (e.g., sequences ofphysical movements, pieces of music). such events may require as muchinterpretation as is required to understand verbal sequences'Technological advances have enhanced our ability to treat verbalprotocols as hard data. Until tape recorders were generally available, itrras common practice for experimenters to take selective notes of ver-balizations, paphrasing and omitting whatever was "unimportant." Inanalyzing such notes further, it was impossible to distinguish the in-ferences from the original verbalizations. using encodings of verbalprotocols as data has often been made even more difficult because theiheories employed, explicitly or implicitly, in the encoding were for-mulated in very general terms. The search for general mechanisms alsoled to overall interpretations of entire protocols with little concern for en'coding and explicating individual protocol statements'More recent research based on explicit information processingmodels of the cognitive process has caused thinking-aloud verbalizationsto be viewed in a new light. It is now standard procedure to make carefulverbatim transcripts of the recorded tapes, thus preserving the raw datain as "hard" a form as could be wished. At the same time, informationprocessing models of the cognitive processes provide a basis for makingihe encoding process explicit and objective, so that the theoretical presup'positions eniering into that process can be examined objectively.

Theoretical PresuPPoClyde Coombs, in hisiriougtt a tYPical sequto tfre eOite and encomake predictions. Thlhrory, can be seen ioitrer *inos of data'ihe universe of Potenjudgment of relevancit ittt next step' thedetermined on theoreIn the case orecording, containingduring the experimentranscript, some selerepetitions, and stressheam, most of thisexcept as it is caPturesteP as PreProcessing'At the next steterminologY of thedetermining codingmake the coding assan indePendent datuon the basis of the inrounding segmentslength methods for cditions that must be

Verbal ProtocoOne method claimsother does requireperimenter have agrmay be sPeech signasignals are mostlYcommunication is Ptween subject and eunder these condispeech signal into oPractice, a code


9/67

though being possibly irrelevanthowever, an analyst .0., ;i;;-.',ot:I. fxation," ,o.ny pry.h"ld;;,a datum but a subjective lnterpta-description of the dream). ;;;,to derive the encoding from teas .,soft', to the degree ttrrrfrry iilwhen the theoretical pr"*rrr'unOco-mpletely explicit un oUroti"eldifferent interpreters ;.*-*;.encodings, and each i"t;;";j;an interpretaton that s favoraUte toto the distinction betweenof infet;;:" ::r'ff::: iSuch events may require as muchverbal sequences.our ability to treat verbalYere senerallv available, to take selective notes of vlr-whatever was .,unimpori.nr.;,'iimpossible to distinguish it, r_Using encodings of u.rUuteven more difficult becuurrth;in the encoding *"r, foi-for general rn..unrrn, uLowith little concer" fb;;;:statements.explicit information processine

thinking-aloud verbalizationiprocedure to make carefulthus preserving the ,." At the same time, inforrnatoiprovide a basis for makingso that the theoretical o;;r";e examined objectively. i


Theoretlcal Presuppositlon in EncodingClyde Coombs, in his book A Theory of Dan, shows that raw data gothough a typical sequence of steps on the route from initial observationto the edited and encoded form in which they are used to test theories ormake predictions. These steps, which are not neutral with respect totheory, can be seen in the processing of protocol data as they can withother kinds of data. At the frst step, theory delimits a small portion ofthe unverse of potentially observable behavior as being relevant. Thisjudgment of relevance determines what behaviors should be recorded.At the next step, these behaviors are encoded in a manner that is againdetermined on theoretical grounds.In the case of verbal behavioi, the process begins with tape-recording, containing essentially all the auditory events that occurredduring the experimental session. In producing from the tape a writtentranscript, some selection is required. After the temporal information,repetitions, and stress have been used to segment and parse the verbalstream, most of this information is usually eliminated from the transcript,except as it is captured by punctuation. we will refer to this transcriptionstep as preprocessing,At the next step, the preprocessed segments are encoded into the(terminology of the theoretical model. This is often achieved by firstdetermining coding categories, a priori, and then having human judgesmake the coding assessments. ff each of the segments is to be treated asan independent datum, then the encoding of that segment must be madeon the basis of the information contained in it, independently of the sur-rounding segments. [n chapter 6 of this book, we will discuss ar somelength methods for carrying out this kind of local encoding, and the con-ditions that must be met to make it possible.Verbal protocols have been analyzed in two rather different ways.One method claims not to require the analysis of meanings, while theother does require it. In the first kind of analysis, subject and ex-perimenter have agreed, by prior instruction, upon specifc signals, whichmay be speech signals or button presses, for their communication. Thesesignals are mostly arbitrary-a subject could say ,.cef" instead of ,,yes";communication is possible only because of the agreement established be-t$'een subject and experimenter. To analyze the recorded verbalizationsunder these conditions, the experimenter has only to categorize eachspeech signal into one of the agreed-upon categories. In theory, if not inpractice, a coder should not even need to know the subject's


10/67

6 Protocol Analysislanguageassuring that no meaningful analysis of inferencing is.involved4A large number "r p.rug*s in psychology use this kind of analysis. Forexampte, studies uril;-i .-l* ij'murtipie-choice alternatives can all be$;;;t nstances of this method' r . . r -.^-L^,iaa 'In the second -io or analysis, the observed verbalizations are

Inferrlng Thought P- r- onrttimes bet' .f^:;'l;i"roretatiosuuJo'li *,,ot -has it'i::- In order to?:::".d for many]]"'"tt"tttlv acceP::^:;;;. the salesl'^lLi-^t least ourTl"i]"i ^,r"ice the,otptt ulterior moi"v review affi;- scientific ;i frnding cit"' review oft'"-obi"ts' reP

,"ise stigtrttY diffe'rl'i*rntt, subjecnto.tttes' T"iJ exPerienctiito""t is nolhat under nufneHowever'think, shoutd) biiit ,ntt X is ti-,tt" informpuit, *t can emucit suUjects nitt. fot the mofrcult it becom'r'Jon"t adve*oi"r that doeConsiderexPerimenter a1' Do You2' WhichcaPital3' Name

iti til ,n "nt concepts mav be verv large' it is severelv'rimited compared "ith ih; variabitity of naturar ranguage. Thus a ver'balizationlike..redcirclesarecefs,'cannormallybeencodedasidenticalit, "Utood-colored round ones are cefs'"The context of a particurar theory and experiment

greatly constrainsthe range of possibtei""rpri.ti"n und uiloo,s the meaningful analysis ofverbalizations to be ."..tiu" and incomplete. If a theory of concept at-hinment is limited ;;'ih; lunguug, of hypotheses, manv verbalizationswill not be encoded ut utt-rt"trnnts like, "[ wonder.what f should do'I'll just guess on this one'" Many exampi" '"n Ot.tlll:t,.:T* kind ofmeaningfulanalysis,whereverbalizationsaremappedontoaprioriformalalternatives. Ttre analysis of memory for meaningful text has beenstudied bv Kintsch-ai'et and manv..otltlt: , *::ll,::1."t-t::

analyzed in terms of ttrJ meanings' Even in this case' the theory buid'ing the analysis timits ltre encodin to "tttt"J u'pt'o and features ratherithan the full meaning i,t r verbilization. For example, in a typical con'rcept attainment task, au'tt intonte or stimulus can be represented as aunique combination oiiruior"r. Each distinct concept can be representedby some particular .;;ltg";;ti"n of features' Then encoding simply re'quires the *upping -l the verbalizations onto these concepts andfeatures usually a rather unequivocal matter' Although *: t:tii.""t:

(1972) analyzed tasts, -ieitifying formally deflrned knowledge states interms of which subjects' thinkine-bud protocols could be encoded'Many analyses of verbatizations do notfit the above scheme' includ'ingmostanalysesthatseektoarriveatanunderstandingoftheverbaliza.tions. In less ror*.i [inos of analysis, the encoding scheme is notdefrned formalty ."d; ;;;;i, but the-search for interpretations proceedsin parallel with the search for an appropriate model or theory' We recog'nize clearly tt. neeJroi and value of such interactive processes in thesearch for theories ii i"* oruins, but in our own acount here we willbe concerne primaiity with situations where the theoretical terms arefixed before the actual encoding begins'


11/67

analysis of inferencing is involved.use this kind of analysis. Foralternatives can all bethe observed verbalizations areEven in this case, the theory buid-

selected aspects and features ratherexample, in a typical con-stimulus can be represented as adistinct concept can be representedThen encoding simply re-onto these concepts andmatter. Although the space ofmay be very large, it is severelyof natural language. Thus a ver-normally be encoded as identicaland experiment greatly constrainsallows the meaningful analysis ofIf a theory of concept at-hypotheses, many verbalizationslike, "I $onder what I should do.can be cited of this kind ofare mapped onto a priori formalfor meaningful text has beenothers. Newell and Simondefned knowledge states inprotocols could be encoded.not frt the above scheme, includ-an understanding of the verbaliza-the encoding scheme is notsearch for interpretations proceedsmodel or theory. We recog-such interactive processes in thein our own aocount here we willthe theoretical terms are


Inferrlng Thought Processes From BehaviorIt is sometimes believed that using verbal data implies accepting thesubjects' interpretation of them or of the events that are reported. Thisissue of trust has its origins in our everyday experience and use of lan-guage. In order to communicate effectively with other people, we acepttheir word for many facts. If someone says that he has bought a neu, car,we generally accept his statement as true instead of asking him toproduce the sales contract or a receipt. In a similar vein we trustpeopleat least our friendsto answer questions correctly and to give usthe best advice they can. However, if the issue is important to us or wesuspect ulterior motives in the responses, we may demand more detailsand may review all the available evidence ourselves. The same thingholds in scientific research; few scientists will accept another scientist'sclaim of finding conclusive evidence for ESP without wanting an inde-pendent review of the evidence.Subjects' reports of their own mental states and mental processesraise slightly different issues of trust. According to a naive theory of con-sciousness, subjects have the sole direct access to their own mental statesand processes. The subjective feeling of one's ability to report one's ownmental experiences veridically is strong. For a great many reasons, thisconfidence is not shared by experimental psychologists, who have shownthat under numerous circumstances such self-reports are unreliable.However, the issue of the reliability of self-reports can (and, wethink, should) be avoided entirely. The report 'X" need not be used toinfer that X is true, but only that the subject was able to say 'X"-(i.e.,had the information that enabled him to say 'X.') By following thispath, we can even show that there is an inverse relation between howmuch subjects need to be trusted and how much information they verbal-ize. For the more information conveyed in their responses, the more dif-ficult it becomes to construct a model that will produce precisely thoseresponses adventitiouslphence the more confidence we can place in amodel that does predict them.Consider, for example, the following possible interchanges betweenexperimenter and subject:l. Do you know the name of the capital of Sweden? Yes.2. Which of these three, Oslo, Stockholm, or Copenhagen, is thecapital of Sweden? Stockholm.3. Name the capital of Sweden. Stockholm.


12/67

I Protocol Analysis4. (A ^retrospecti.ve report as to how the subject arrived ar an answerto Question r): ,Frst I. tried to picture where sweden rocated on amap of Europe, then osro came mind, but L,r*^iri that it is thecapital o Norway. Then snckhorm popped up and I renimbered ilatis where the Nobel prizes are awarded;'in"n *if*n-'coutd answer"ygs,"In the first case we have to trust the subject if we want to infer thathe actually knows the capitar, whereas in the third case it is untikery thathe could generate the correct name unless it were accessible frommemory. The primary difference between second and third cases is that,for the second, one courd conceive of a number of processes other thanmemory retrieval (e.g., guessing) that would ur.ount foi-tfr" response.The fourth response, the retrospective report, also verifies that the sub-ject has the name in memory together with some redundant-informationabout it that gives him confidence in his answer. of course we do nothave to believe that he has given a veridical report of the processwhereby he generated the name, arthough there is nothing imprausibleabout the sequence ofassociations he reports.Consider next a more controversial example, which has played aple in the psychological literature on learning without awareness. After alearning experiment, the experimenter asks the subjects "*rrlrr'.,ere aware of any relation between the stimuli uno ,"rpons"r, fn the onehand, and the reward contingencies on the other. yes/no responses tothis question are informative onry if we trust the subjects. If a subject,however, describes the stimulus-response contingency for reward, we canbe reasonably certain that he had'access to this information while he waslearning. on rhe orher hand, if a subject is unabre " *ori-.irir,,r,gabout the contingency, we cannot conclude that he wasn,t aware of itduring the learning processwe have sorid evidence neither for noragainst awareness during the experiment. Later, we will iscuss theproblem of making inferences from reports of rack or nror*utin.These exampres ilrustrate ttrat the information externarized in verbarresponses often provides the experimenter with data that eliminate theneed for trust in the subject. The examples arso show that verbal reportsmav be generated in many ways. To understand the *-t,, we mustunderstand the proces-ses by which they were generated. In none ofthese respects do dat from verbar reports differ from oata uase on othertypes of observations.

Some Baslc AssumPtlonsWe cn now summarize the bfurther explorations. Most funtype of recordable behavior, wany other behavior. The cognare a subset of the cognitive pable response or behavior. He.,mechanical" and complete proother kinds of behavior, and wprocesses as explanations for vetilhether one can and shomatter of faith but an empiricalother types of behavior, like einvalid verbal report should nreports generally. Indeed, th

theory of verbalization, so that wder what kinds of instructions infrom subjects.Postulating that the cognita subset of all cognitive processwith the constraints that have ball cognitive processes. Theseprovide powerful guidelines for obalizations could have been genreported information by proposinful to regenerate that informationPlan of Attack

Our frst task is to describe a gstructure, which, we argue, accouFor reasons that have already bouf within a framework of theogeneral to permit us to relate, wof data that are commonly used iUsually, in choosing betwstrongest on+the one that will


13/67

subject arrived at an ans$,erpicure where Sweden is located on amind, but I remembered tltat it is theup and I remembered tlattlrcn I felt sure I could answerthe subject if we want to infer thatin the third case it is unlikely thatunless it were accessible fromsecond and third cases is that,a number of processes other thanwould account for the response.also verifies that the sub-with some redundant informationhis answer. Of course we do nota veridical report of the processthere is nothing implausible

example, which has played awithout awareness. After aasks the subjects whether theystimuli and responses, on the oneother. Yes/no responses totrust the subjects. If a subject,contingency for reward, we canto this information while he wasis unable to report anythingthat he wasn't aware of itsolid evidence neither for norLater, we will discuss theof lack of information.information externalized in verbalwith data that eliminate thealso show that verbal reportsunderstand the reports, we mustwere generated. [n none ofdiffer from data based on other

Introduc{ion and Summary 9

Some Basic AssumPtionslVe can now summarize the basic assumptions that set the stage for ourfurther explorations. Most fundamentally, we see verbal behavior as onetype of recordable behavior, which should be observed and analyzed likeany other behavior. The cognitive processes that generate verbalizationsare a subset of the cognitive processes that generate any kind of record-able response or behavior. Hence, we would look for the same kind of"mechanical" and complete process description of verbal behavior as ofother kinds of behavior, and we would not accept magical or privilegedprocesses as explanations for verbalizations.Whther one can and should trust subjects' verbal reports is not amatter of faith but an empirical issue on a par with the issue of validatingother types of behavior, like eye fixations or motor behavior. A singleinvalid verbal report should not force us to discard analysis of verbalreports generally. Indeed, this monograph will undertake to build atheory of verbalization, so that we can then specify when, where, and un.der what kinds of instructions informative verbal reports can be obtainedfrom subjects.Postulating that the cognitive processes underlying verbalization area subset of all cognitive processes implies that verbalization must complywith the constraints that have b.een identified, experimentally, to governall cognitive processes. These information processing constraints willprovide powerful guidelines for our attempts to specify how observed ver-balizations could have been generated. We wish to account for verballyreported information by proposing a processing model sufliciently power-ful to regenerate that information.Plan of Attack

Our first task is to describe a general theory of cognitive processes andstructure, which, we argue, accounts for verbalizations and verbal reports.For reasons that have already been stated, the analysis must be carriedouf within a framework of theory. This framework must be sufficientlygeneral to permit us to relate, within a unifred perspective, all the kindsof data that are commonly used in psychological experiments.Usually, in choosing between theories, we want to pick thestrongest ono-the one that will make the strongest predictions. In the


14/67

specific hypotheses where they are required'- After presenting the theory as an information processing model ofcognitive processes, we will survey the literature on verbal reporting anddeiive from it a taxonomy of reporting procedures. We will follow thissurvey with an historical review of earlier approaches to verbal feports.we will then take up the major issues surrounding the use and validity ofverbal reports, discussing the empirical studies within the framework of amore detailed information processing model'THE PROCESSING MODELour purpose in presenting a specific processing model is to aid us in in't.rprting verbal data obtained from subjects and the relation of theirverbal to their other behavior. Since the data (including the verbal data)are gathered in order to test theories about the human informationprocssing system, we are engaged in something of a bootstrap operation.we need a model in order to interpret data that are to be used, in turn,to test the model. under thse circumstances, our data-interpretationmodel should be as simple as possible, and it must not incorporate com-ponents that are themselves bones of theoretical contention. The modelstroutO be robust (i.e., compatible with a wide range of alternative as'sumptions about human information processing)'Thespecificationsweareabouttopresentaresimpleandrobustinthis sense, and, indeed, summarize the core that is common to most cur-rent information processing theories of cognition. of course they are notentirely neutral, for they would be hard to reconcile with an extremeform ofbehaviorism that denied the relevance of central processes to theexplanation of behavior. But they are not specific to the view of any par'ticular "sect" within the general information-processing tradition' (Forfuller discussion of the model, see Newell and simon Q972, Chapter 14),and Simon (1979, ChaPters 2, 3).

Genersl SPeciflcationThe most general and wnition is inlbrmatton Pr,rqotntt of internal^ strtion pfocesses.

Aninformution is stored.inaccessing characteristicsinort-term memory (duration, and a long-trelatively Permanent smpared with the otherVithin the framesumed that informationntral Processor is keorocessing (e'g., for PLTM tutt first berePorted.This general Phypotheses that havemechanisms. For ex"short'term memorYportion of LTM that19?6). Some theoristspassage of time, unieplaced. In general,at the level of sPechypothesis for us ismost recentlY heededtion of the contentsSTM, and this Porretrieved from LTM'Our sPecifrcatioSpeciflrc informationhave been constructe*Because the Phrase "atiti-"iu- t eeded" inili;;ii"" nas "atten


15/67

will influence the way in whichwnt to pick the weakest and mostfewer controversial assumptionswe will be involved in the cir.test our theories. Nevertheless,data that does not incorporatethe system and processes thatwill be to set forth the theoryform first, then add moreinformation processing model ofliterature on verbal reporting andprocedures. We will follow thisapproaches to verbal reports.surrounding the use and validity ofstudies within the framework of a

model is to aid us in in-subjects and the relation of theirdata (including the verbal data)about the human informationof a bootstrap operation.data that are to be used, in turn,our data-interpretationand it must not incorporate com-contention. The modela wide range of alternative as-present are simple and robust incore that is common to most cur-Of course they are notto reconcile with an extremeof central processes to thespecific to the view of any par-tradition. (Forand Simon (1972, Chapter 14),

Introduction and Summary llGeneral SPecification

The most general and weakest hypothesis we require is that human cog-ntion is information processing: that a cognitive process can be seen s asequence of internal states successively transformed by a series of infor-mation processes, An important, and more specific, assumption is thatinformation is stored in several memories having different capacities andaessing characteristics: several sensory stores of very short duration, ashort-term memory (sTM) with limited capacity and/or intermediateduration, and a long-term memory (LTM) with very large capacity andrelatively permanent storage, but with slow fxation and access timescompared with the other memories.Within the framework of this information processing model, it is as-sumed that information recently acquired (attended to or heeded)' by thecentral processor is kept in STM, and is directly accessible for furtherprocessing (e.9., for producing verbal reports), whereas information fromLTM must frst be retrieved (transfered to STM) before it can bereported.This general picture is compatible with all sorts of specifichypotheses that have been put forth with respect to the details of themechanisms. For example, some theorists propose that what we call"short-term memory" is not a separate, specialized store but simply. aportion of LTM that is currently and temporarily activated (Anderson,1976). Some theorists believe that information in STM extinguishes withpassage of time, unless rehearsed; others that it is lost only whenreplaced. In general, these differences of detail do not affect the modelat the level of specifrcity required for our purposes. The importanthypothesis for us is that, due to the limited capacity of STM, only themost recently heeded information is accessible directly. 'However, a por-tion of the contents of STM are fixated in LTM before being lost fromSTM, and this portion can, at later points in time, sometimes beretrieved from LTM.Our specification of the system is general, but it is not vague.Specific information processing models that incorporate these featureshave been constructed in the form of computer programs, and these have'Because the phrase "attended to" is often stylistically awkward, we will some-times use "heeded" instead. So we will say, more or less synonymously, thatinformation was "attended to," was "heeded," or was "stored in STM,"


16/67

12 Protocol Analysisbeen shown to produce a variety of behaviors previously observed inpsychological laboratories. verbal predictions of how such a system be.haves can, thereby, be tested by using a computer program as asimulator. The principal model of this kind that guides our own thinkintabout these processes is the EpAM program, due to Feigenbaum (1963iandSimon,anddiscussedinsomedetailinSection3ofSimon0g7g),

we assume that any verbalization or verbal report of the cognitiveprocesses would have to be based on a subset of the information held insrM and LTM. From this and the above hypotheses, the taxonomy ofverbalization procedures shown in Table l-l foltows in a straightforwardfashion (Ericsson & Simon, 1980).Table 1-lA classification of Different Types of verbalization procedures as a Function ofTime of verbalization (Rows) and the Mapping From Hedel o veiualzeo tn_formation (Columns)

Relaton between heeded and verbalized rnformatlonIrtcrmdiate proccssrng

Time of verbalizatron )irect one to one Many to one No rclatronWhile nformatron sattendcdWhile tnformaton sstill rn short-termmcnrofyAfter the completlonof the task-directedProccss

The two dimensions of rable l-l represent two major distinctions.First, the time of verbalization is important in determining from whatmemory the information is likely to be drawn. second, we make a dis-tinction between procedures where the verbalization is a direct articula-tion or explication of the stored information, and procedures where thestored information,is input to intermediate processes, like abstraction,andinference, so that the verbalization is a product of this intermediateprocessing.

Talk aloudThrnk aloudConcurrentprobtngRetrospcctlvprobrng

Intermediate rnference and gcncfattve proccsesRequests for rrobrng ProbrngSenefal reports hypothctlcal gcneral statcsstates

Detiled SPeclflcatlo\IVe now specify moresystem that we havesources that are sumSimon 0979, Ch.2.3sial. It makes no resingle homogeneousAnderson, 1976; Shifstores (sensory storecan be described in eformation that can bereporting them verbamemories in our descRecognition. Infor a short time inwith the different senmation are directlY reready stored in LTMof it, with existing pthose familiar patternrecognition mechanprocess (the successivto 100 msec, do not uLong-Term Mecollection of interrelanition (through thelinks that associate thlnformation accessedSTM. Thus, informavia the recognitionAsspciation prooessesrequiring.at least sesociative processes mample, in recalling amay use a sequencethe sought-for nameminutes, and may lethey are temporarily


17/67

Introduction and Summary 13of behaviors previously observed inpredictions of how such a system be-by using a computer program as athis kind that guides our own thinkingprogram, due to Feigenbaum (1963)detail in Section 3 of Simon lJ97.

or verbal report of the cognitiveon a subset of the information held inabove hypotheses, the taxonomy ofTable l-1 follows in a straightforward

Verbalization Procedures as a Function ofMapping From Heeded to Verbalized In.

betwn heeded and verbalized tnformatonIrtcrnrediate proccssrng

Many to one No rclatron

Detailed SPeclficationwe now specify more.fuily the components of the information processingsystem that we have just sketched. The model draws upon a variety ofsources that are summarized in Newell and simon (tgiz, ch. 14) andsimon 0979, ch. 2.3). Few of the model's specifcations r, .on,.ourr-sial. It makes no real difference, for exampre, whether *L urrurn usingle homogeneous memory with different modes of activation (e.g.,Anderson, 1976; shiffrin_& schneider, 1977) or severar di;;;;. memorystores (sensory stores, srM, and LTM). The important maite.s, whichcan be described in either terms, relate to the amunts and kinds of in-formation that can be retained, and the conditions ro, u..rrrJn them andreporting them verbally. we will use the conventional modelf multiplememories in our description.Recognitron. Information received from the sensory organs residesfor a short time in memories (iconic and echoic ,r*r"ri'associatedwith the different senses, During this time, portions of the sensory infor-mation are directry recognized and encoded with the aid of information ar-ready stored in LTM. Recognition associates the stimulus, or some partof it, with existing patterns in LTM, and stores n irrr'.;pointers,, tothose familiar patterns. (rne gperu discrimination net is a model of thisrecognition mechanism.) Intermedite stages of the direct recognitionprocess (the successive steps of discriminatin), *rrrct, ;;;irk, onry r0to 100 msec, do not use STM to store their products.Long-Term Memof. The LTM may be pictured as an enormouscollection of interrerated nodes. Nodes can be accessed either by recog-nition (through the discrimination net), as just exprained, or by way oflinks that associate these nodes to others that have already been accessed.Information accessed in either way is then represented by pointers insrM. Thus, information can be brought into srM from sensory stimurivia the recognition process, or from LTM via the association process.Asspciation processes are much srower than direct recognition processes,requiring.at least several hundred msec for each associative step. As-sociative processes may use srM to store intermediate steps. so, for ex-ample, in recalling a name that is not immediately accessibrr,'u personmay use a sequence of cues to find an associative path, step y step, tothe sought-for name' such processes may rast tens or,.*nd., or evenminutes, and may reave numerous intermediate symbors in srM, wherethey are temporarily available for verbal reports.

Intermediate rnference and gcncratrve proccsesRequests for I'robrng lrrobrnggeneral reports hypothctrcal gcneral stategstates

1-1 represent two major distinctions.important in determining from whatto be drawn. Second, we make a dis-the verbalization is a direct articula-and procedures where theprocesses, like abstraction,andis a product of this intermediate


18/67

14 Protocol Analysisshort-term Memory. The central pfocessor (cP), which controls andregulates the non-automatic cognitive processes' determines \4'hat smallput of the information in s"n.oty stimuli and LTM finds its way intobfU. fnir is the information that is heeded ot attended fo. The amountof information that can reside in STM at one time is limited to a smallnumber (four?) of familiar patterns Ghunks). Each chunk is rePresentedby one symbol or pointer t information in LTM (Simon 1979, Ch.2.2).A, nrr" information is heeded, information previously stored in STMmay be lost.when a cognitive task (e.g., mental addition of a column of fgures) 'isbeingcarriedout,thetypicalchunksinSTMarepointerstotheoperands, operators, and outputs of the operations that are being per'formed. Thus, in adding 3 t 4, pointers corresponding to the symbols;i;; ..4," ..pLS," und "?" might at some time be present in STM.since, in our culture, adding two digits involves a direct reference toLTM (.'table lookup'), no further detail of the process would be heededin STM or available for verbal reports' On the other hand, if the tskweretomultiplylTby45,STMmighthold,atvariouspointsintheparar, ,r45,r, ,r17,u,.7-" ..'1yBS," "3" (the Carfy in mUltiplying 45 byif, *315' (the frst intermediate product), '45," "1," ',TIMES,"*PLUS,"',765."we hold no brief for the details of the above description, which isintended merely as an example of the kinds of information we would ex-pect to be heeded in sTM, and to be available, potentially, for concurrentr retrospective reports. The speciflic details would depend on the par-ticutar strategies subjects used and the nature of the chunks they had;il in r,irr,r (Simon, lg7g, Ch. 2.4\' STM would svmbolize theprocess only down to some modest level of detail (corresponding toelementary processes of a second or two in duration), and we would notexpct to nno information there about simple, automated

processes (e'g',th processes of retrieval from LTM or recognition processes),. much lessabout neuronal events. Thus, the architecture of the control apparatus(cP) determines the fineness of grain of the representation of processesin STM.

control of Attention. The flow of attention is diverted, from timeto time, by interruptions through the higher control mechanism. Inter-inediate stges in these interruptions, not being symbolized in STM' arenot reportable. sudden movements in peripheral vision, loud noises,

emotions operatinglerruption and shaiion treeoed immtimes allow subjectswe woutO exPectsuotttY Process tha(e.g., a thought sethought Processes''

Fixstion. NeCP is attending toinformation that ccoding and imaginnetwork. These lmation in LTM anet for accessing iSection 3). Procesemble each newstore it in LTM asAutomationbecome more anAutomation meaninterpreted, andautomation of Pecomputer algorithtion (and comPspeed uP the Procthe intermediateverbal reports.

TYPES OF VEThe only featuretain verbal dataprobe. Becauseto the Probes wwill elicit someWithin ou


19/67

(CP), which controls andprocesses, determines what smalland LTM finds its way intoheeded or attended to. The amountat one time is limited to a smallEach chunk is represented

in LTM (Simon 1979, Ch.2.2).previously stored in STMaddition of a column of figures)in STM are pointers to thethe operations that are being per-corresponding to the symbolsat some time be present in STM.involves a direct reference toof the process would be heededOn the other hand, if the tskhold, at various points in the

3" (the carry in multiplying 45 byproduct), "45," "1,tt ."TIMES'"of the above description, which iskinds of information we would ex'potentially, for concurrentdetails would depend on the par-nature of the chunks they had2.4). STM would symbolize thelevel of detail (corresponding toin duration), and we would notautomated processes (e.g.,recognition processes), much lessof the control apparatusof the representation of processesof attention is diverted, from timehigher control mechanism. Inter'not being symbolized in STM, arein peripheral vision, loud noises,

Introduction and Summary l5emotions operating through the reticular system are important causes ofinterruption and shift in attention (Simon, 1979, Ch. 1.3). While infor-mation heeded immediately before or after a shift in attention may some-times allow subjects to give a relatively clear account of the interruption,we would expect such information to be less complete than reports of anorderly process that is induced by the successive content of STM itself(e.g., a thought sequence during which goals in STM are guiding thethought processes).

Fixation. New information is retained in STM during the time theCP is attending to it. [n order to create an LTM representation of newinformation that can later be recalled, associations must be built up bycoding and imaging, as well as new tests and branches in the recognitionnetwork. These learning processes, including the storage of new infor-mation in LTM and the addition of new pathways in the discriminationnet for accessing it, are modeled in some detail by EPAM (Simon, 1979,Section 3). Processing of the order of 8 to 10 seconds is required to as-semble each new chunk from its familiar components in STM, and tostore it in LTM as a new chunk (Simon, 1979, Chs. 2.2,2.3).Automation. As particular processes become highly practiced, theybecome more and more fully automated. (Shiffrin & Schneider, 1977),Automation means that intermediate steps are carried out without beinginterpreted, and without their inputs and outputs using STM. Theautomation of performance is therefore quite analogous to executing acomputer algorithm in compiled instead of interpretive mode. Automa-tion (and compiling) have two important consequences. They greatlyspeed up the process (typic.ally, by an order of magnitude) and they makethe intermediate products unavailable to srM, hence unavailable also forverbal reports.

TYPES OF VERBALIZING PROCEDURESThe only feature common to the whole range of techniques used to ob-tain verbal data is that the subject responds orally to an instruction orprobe. Because of the flexibility of language, there are virtually no limitsto the probes we can insert and the questions we can ask subjects thatwill elicit some kind of verbal response.Within our theoretical framework, we can represent verbal reporting


20/67

16 Protocol AnalYsisas bfinging information into attention, then, when necessary' convertingit into verbalizable code, and finally, vocalizing it. The crucial issue fsrverbal reporting procedures is what information is heeded' There hatebeen studies showing that the response modality does not affect the fre;;;;t oi different i6ponr.r. Newhall and Roderick (1936) found noiiff.rn.tt in frequencies between verbal reports, button presses withfngers, or pedal presses with the feet. This result indicates that the,..ponr. is heeded symbolically, and then translated into the appropriateovrt form. (See Chapter 5 for further discussion')Two forms of verbal repofts cn claim to being the closest reflectionof the cognitive processes. Foremost ale concurrent verbal reports'"talkaloud" and "think aloud" reportwhere the cognitive processes'described as successive states of heeded information, are verbalized

stotos Of He

directly (see Figure l-l).We claim that cognitive processes are not modified by these verbalreports, and that task-directed cognitive procsses.determine what infor'*tion is heeded and verbalized. We will evaluate this claim empiricallyin Chapter 2,A second type of verbal report is the retrospective report. A durable(if partial) memory trace is laid down of the information heeded suc'ces'sively while completing a task. Just after the task is ftnished, this trace."n b. acessed from sTM, at least in part, or retrieved from LTM andverbalized. Retrospective repofts based on information in LTM will re-quire an additional process of retrieval that will display some of the same[inos of error and incompleteness that are familiar from experimentalresearch on memory. Both of these kinds of reports, we claim, are directverbalizations of specific cognitive processes.Recoding Before Verblizationvarious processes, and especially recoding processes, may intervene be'tween the time information was heeded by the central processor (CP)and the time a verbalization is generated. when information isreproduced in the form in which it was heeded, we will speak of direct orLvet l verbalization. When one or more mediating processes occurs be-tween attention to the information and its delivery, 'we will speak ofencoded or Level 2 0r Level S verbalization. A number of different kindsof intefmediate processes between acess and verbalization may modifythe information. Among the important kinds are the following:

rorkAroud f',',-l-L-L,tVocollzotlon(l )

Thrnk Aloud

Vorbollzolion ProVerbolizotioFTFlgure 1-1The Relation BetweeReports for Various T

Vocatlzollo ( t )


21/67

then, when nec.essary,finally, vocalizing it. The crucial issuewhat information is heeded. Thereresponse modality does not affect theNewhall and Roderick (1936) foundverbal reports, button pressesthe feet. This result indicates thatand then translated into thefurther discussion.)can claim to being the closestate concurrcnt verbal reportreportswhere the cognitiveof heeded information, are

processes are not modified by thesecognitive processes.determine whatWe will evaluate this claimreport is the retrospective report. Adown of the information heededJust after the task is finished, this traleast in part, or retrieved from LTMbased on information in LTM will re-that will display some of the samethat are familiar from experimentalthese kinds of reports, we claim, are directprocesses.

recoding processes, may intervene be-heeded by the central processor (Cp)is generated. When information isit was heeded, we will speak of direct oror more mediating processes occurs be-and its delivery, we will speak ofA number of different kinds.aocess and verbalization may rnodifykinds are the following:

Introduction and Summary l?Stol6s Of Heeded Informotion In A CognilvE process

Vocolzotion(3)

Tot[ Aloud

Thtnk Aloud

vocqrzolontt, vocoliqllon(e) Vocollzofon(glVerbollzofion procedures Thol Involvs Modiolng processes BefoeV0rbolzoton, Liks Requesfs For Explonolions, Motions elc.

Flgure 1-lThe Relation Between the tl-eeded states of a cognitive process and verbareports for various Types orviiR Lcedures

Vocoll zotlon ( Z)

Vocollzotlon t t I


22/67

[8 Protocol Analysis1. Recoding into verbal code (Level 2 verbalization). when the in-,ternal representation in which the information is originally encoded is nota verbal code, it has to be translated into that form. werner and Kaplan(1963) have shown that when subjects generate verbal descriptions ofnonverbal stimuli for their own future use, the format is compact and in'corporates many idiosyncratic referents. v/hen verbalizations pfe

generated to communicate information to another person' additionalrocessing is required to find referents (Werner & Kaplan, 1963).2. Intermediate scanning or filtering processes (Level 3verbalization). When the task instructions ask for verbalization of onlyselected information, it is necessary to postulate additional processes thattest if the heeded information is of the desired type. such instructionsare used, for example, in commentary drivig experiments, in which thesubjects are asked to report all perceived traffrc hazards while they aredriving a car (Soliday & Allen, L972r.t Intermediate inference or generative processes (Level 3

additional informamation.RetrosPectlve ReP

In the ideal casemediatelY after thSTM and can bethat some additionparticular memorythat retrosPectivecn be accessed anthe subject with sticular case, the sSTM when a generemember about yretrospective verbthe actual memoryEven in thisto all kinds of verfallible, in that othof those created bthis occurring increof similar problemcontain redundantjects may use thisas to increase themation. [n a subsfurther and examinA second genseparate informatiofrom informationwith it (Mueller, lfriend, it may be teto i4fer what the pdiscuss this issue itifact by instructingheeding at the tima requirement of d

verbalization) The situation is even more complicated if the ex-perimenter is interested in particular aspects of the situation that a subjectwould not ordinarily attend to. The issue of whether the instruction toverbalize calls for information not normally heeded by the subjects iscentral and directty related to the occurrence of intermediate inferenceand generative processes. since we will return to this issue in moreOepth, only a brief summary will be given here of the types of infor-mation that are likely to require additional mediating processing for theirgeneration.In addition to verbalizing their ongoing thinking, subjects are some-times asked for verbal descriptions of their motor activities, for example,what objects are moved where, or where they are looking' When thisinformation is not heeded directly, as is often the case, the subject is re'quired to observe his or her own internal processes or overt behavior togenerate the information.Experimentersareofteninterestedinsubjects'reasonsfortheirovert behavior and consequently ask the subjects to verbalize their mo-tives and reasons, which,may not be available directly or even at all. [nan excellent review of research on the effects of persuasive messages'wrieht (19S0) discusses a wide range of biases due to different verbalreport procedures.- In sum, with Level 1 and Level 2 verbalization the sequence ofheeded information remains intact and no additional information isheeded. on the other hand Level 3 verbalization requires attention to


23/67

2 verbalization). \lVhen the in-is originally encoded is notinto that form. Werner and Kaplangenerate verbal descriptions ofuse, the format is compact and in-When verbalizations areto another person, additional(Werner & Kaplan, 1963).filtering processes (Level 3ask for verbalization of onlypostulate additional processes thatthe desired type. Such instructionsdrivig experiments, in which thetraffic hazards while they are

generative processes (Level 3more complicated if the ex-of the situation that a subjectissue of whether the instruction toheeded by the subjects isof intermediate inferencewill return to this issue in moregiven here of the types of infor-processing for their

thinking, subjects are some-motor activities, for example,are looking. When thisis often the case, the subject is re-processes or overt behavior toin subjects' reasons for theirthe subjects to verbalize their mo-available directly or even at all. Ineffects of persuasive messages,of biases due to different verbal

2 verbalization the sequence ofno additional information isverbalization requires attention to


additional information and hence changes the sequence of heeded infor-mation.RetrosPectlve RePorts

In the ideal case the retrospective report is given by the subject im-mediately after the task is completed while much information is still inSTM and. can be directly reported or used as retrieval cues. It is clearthat some additional cognitive processing is required to ascertain that theparticular memory structures of interest are heeded. Our model predictsthat retrospective reports on the immediately preceding cognitive activitycan be accessed and specifred without the experimenter having to providethe subject with specifc information about what to retrieve. In this par-ticular case, the subject will still retain the necessary retrieval cues inSTM when a general instruction is given "to report everything you canremember about your thoughts during the last problem." This form ofretrospective verbal report should give us the closest approximation tothe actual memory structures.Even in this favorable casen some problems arise that are commonto all kinds of verbal reports from LTM. First, the retrieval operation isfallible, in that other similar memory structures may be accessed insteadof those created by the just-finished cognitive process. The probability ofthis occurring increases markedly if the subjects have just solved a seriesof similar problems. However, since most accessed memory structurescontain redundant information beyond the cues used for retrieval, sub-jects may use this additional information to validate the retrieval as wellas to increase their confidence in the veridicality of the retrieved infor-mation. [n a subsequent section we will discuss this type of evaluationfurther and examine the relevant theoretical and empirical titerature.A second general problem when retrieving cognitive structures is _toseparate information that was heeded at the time of a specific episodefrom information acquired previously or subsequently that is associatedwith it'(Mueller, l91l). For example, if a picture reminds one of an oldfriend, it may be tempting to use the stored information about that friendto i4fer what the person in the picture looked like. (ln Chapter 3 we willdiscuss this issue in more detail.) It may be possible to eliminate this ar-tifact by instructing subjects only to report details that they can rememberheeding at the time of the original episode (Mueller, 1911). By imposinga requirement of determinable memory as a basis for reporting, we can


24/67

20 Protocol AnalYsis

avoid many subjects' tendency to frll in information that theyi"*"*U"t Lut "must" have thought'

Themostmarkeddifferencebetween.concurrentandretrospectiG.reporting is that "n"otU-* "po"t r3f9r to a cognitive process that iscompleted.nd"nnJt""b"-''i1"{,;1'1:1111'L"T::"-'.:lff :l';';

Dhecterl or SpecializVerbal Probes diffe;;tttd and the genilJi"t first considerttttutO only in Pi"Oat ProUe may be,iu,ott tPttifrcallY rert lo nAP subjectsinOo* greatet comporut of,tn contains anatYzing verbal rixed set of alternalint nttuut renortsittt*ttt or are thIn most casereports is not an imbe less, rather tha[no*n to us usingthat subjects have rOne comrnonwhen the subjectsknow that the queirttt ot. emPloyingsubjects don't havsets of alternativeprocessing, and heio the actual thouprobes are used,this has occurred'Since Providaid recall from LTis frequent andrePort on immedduration, specifrccal sense, the exject in the third a(l) Directed ProQuestion:- DAnswer: Ye

::ffJ,:iiJli"i1il:*: :l':,'ver heeded' thev cannot pos' sibly base their responses on dired memory' The subjects can answetthat they don't know,;"i q".t Yitlinfer and generte an ansrter onthe basis of informati* n'"iOtO in the^question and other informationaccessible rrom lrr"rl" in." t"rrieval from LTM may be an onefoustasko even in situatin, i"i" tt " information is potentially retrievablesubjects may prefer to g"nututt the. information instead'rhe most .o*;; o.be jhat':'"::i-Jli.1'.*1"",:t *t"i"i:;0".,,1i H:iJii::1'\;ii:;-o it'i't" or "rrvhv did.vouprerer

that product?" rn u'i'in';; it;"tsion' Lazarsfeld (1935) points tomany issues and p';i;;;-i; ilnterpreting it'ponttt to whv-questions'where subjects ,rt..t-i" alternative out oi senrat possibilities' some ofthe alternatives may i'"o -ftt"t been heeded' If we wish to frnd out:,whydidyoubuythisbook?,,wemayreceive,outofthesameconcreteexperienceortheiJ"i"ll;,::':"::f:"ilr1i:",'"":ir"J,o.J:ffiilr'irli** ,iffio,u, * long that r shouldn't have got it rot two 1months.' tf tre unaerJtoiwr'v di vor u"t *us uo:^kJ' he mieht tell what interested him *nt"itfft in ttre. littt' And if he understood: 'i.why did you buy ttrir-g-ooKl-' ne migtrt repott,ttua he at frrst thought of Ibuying a concert ti'r"tn irr" *on'y' uuiruttt realized that a book is a much more durable ,t ing it * a concrt, and such reasoning caused himto spend his monev "t;9k"(Lazarsfeld' 1935' p' 29)'rhe ex ampl, r' inrtru.liv e in show ir9 lll,jl::".,T.1; _i

if,i:ii.:irilJdit:'"""1nii:".;33it'"'rr the respondent under- {srood: why did you BuY this book?' li::,**lm ":l:: Jl:

,".,'r"i1i;: ;i'ffi;;il; 'r* **'v or the associated icognitive processes' d.;"','ry 'T: ii'::: fi:-:iJ'i'ur ""

:::#i"'J::'J;i""::'"il;*; ru* Hence' the answers can begenerated (inferred) *'i,i"i .*s to a specifrc memory trace of theepisode.

Inferential or Generative Processes


25/67

fill in information that they can't

concurrent and retrospectiverefer to a cognitive process that isinfluenced. Hence, if subjects arewas never heeded, they cannot pos'memory. The subjects can answerwill infer and generte an answer onthe question and other informationfrom LTM may be an onerousis potentially retrievableinstead.creates this problem is the why'do this?" or "Why did you preferLazarsfeld (1935) points toresponses to why-questions,out of several possibilities. Some ofheeded. If we wish to find out:receive, out of the same concretedifferent answers, according to"book." "If the respondent under-he might answer, 'Because thethat I shouldn't have got it for twoyor buy THIS book?'he might tellauthor. And if he understood:report that he at frrst thought ofbut later realized that a book is aand such reasoning caused him(Lazarsfeld, 1935, P'2.that a person who did not ac'no specific memory of the associatedsame or similar answers as plausiblebook. Hence, the answers can beto a specific memory trace of the

Introduction and Summary 2lDlrected or Specialized Problng

Verbal probes differ in the comprehensiveness of the topics to bereported and the generality or particularity of the events to be reported.Let us first consider topic specificity. In many studies, the investigator isinterested only in particular aspects of subjects' behaviors. Then theverbal probe may be constructed to induce the subjects to generate infor-mation specifically relevant to the hypotheses under consideration. In or-der to help subjects retrieve the desired information from memory and toinduce greater completeness of the verbal reports, the question or verbalprobe often contains contextual information. To guard against subjectivityin analyzing verbal reports, the investigator often supplies subjects with afixed set of alternative responses. In contrast, a general instruction togive verbal reports typically asks subjects to tell everything they canremember or are thinking of while performing the task.In most cases, verbosity and absence of selectivity in subjects'reports is not an important problem. What the subject reports is likely tobe less, rather than more, than we should like to hear. In no studyknown to us using general instructions has the investigator complainedthat subjects have reported too much information from actual memory.One common difliculty in probing for specific information, especiallywhen the subjects are offered a fxed set of alternative answers, is toknow that the questions conform to the internal representations the sub-jects are employing in their thought. Probes for types of information thatsubjects don't have directly accessible, or probes that provide inadequatesets of alternatives may force subjects to intermediate and inferentialprocessing, and hence produce verbal reports that are not closely relatedto the actual thought process. Moreover, when specific, fixed-alternativeprobes are used, there is no way to detect from subjects'responses thatthis has occurred.Since providing contextual information and prompts to subjects mayaid recall from LTM, in studies of LTM the use of prompts and contextis frequent and relatively well-motivated. when subjects are asked toreport on immediately preceding cognitive processes of relatively shortduration, specific probes are more questionable and less useful. In a rogi-cal sense, the experimenter gets just as much information from the sub-ject in the third as in the first two of the following three cases.(l) Directed probe IQuestion: Did you use X as a subgoal?Answer: Yes.


26/67

The replies in all three cases provide evidence that the subject u6X as a subgoal, yet the evidence is stronger in the third case than in thesecond, and in the second than in the first. The verbalization in the fnstcase coul{ easily be generated by processes independent of any memory

22 Protocol AnalYsis(2) Directed Probe 2Quettion: Did you use any subgoals? lf so, whrch?Answer: Yes, I used X.(3) Undirected ProbeVerbal repoit: ...1 was first trylng to get X and l"'when I attaned X,"

Prtlcular nd GenIf the purpose of re',:ffi iil''i::ffi '..rni. fot at least tittt, .fttt a seriesiil*ss6 will be P'Jrnty fot recurrentttt.t it*tt or nonehter trials of the exSecond, manycharacteristics of thitrr nOiuidual tria.,How did you do tsuest a general rawere Performing th

There are sedescriPtions of thspecific behaviorsuia .o'utt of the gessentiallY the samdirectlY withoutAnother Possibiliprocesses duringgeneral Proceduresubjects remembeprocesses used fogeneral Procedurevarious kinds ofone ought to dogeneral Procedurebear anY close reson,1977).ln areas ofhas a long tradrather than for gtechnique ProPosto report their me

for the actual thought processes. Comparing the second and third cases'rthe former communicates to subjects what information the experimenteexpects them to report. It may encourage subjects to try to infer or guesswhat particular information the experimenter will accept, and to generateinformation accordinglY.ln many cases, other criteria are available for estimating the validity r$of the reports. An analysis of the task (Newell & Simon, 1972) will often':provide tttong indications of the adequacy of verbalized information,especially in cases with many logical possibilities for response. ,jiFinally, different kinds of probes may have different effects upon ithe behavior of subjects. Requesting a certain kind of information may suggest to subjects what aspects of the task are important. subjects may also alter their normal ways of processing so as to be able to give the Irequestedinformationtotheexperimenteronsubsequenttrials.In studies that use retrospective verbalization, subjects are seldom 'asked what they can remember about speciftc instances of their cognitiveprocesses. Rather, they are usually asked to retrospect about theirthought processes in experiments with many trials or to answer generalqurrtionr, and thus must try to synthesize all the available informationafter selective recall. In making iudgments, subjects have access to anextremely large base of relevant knowledge' Tversky and Kahneman(1973) have demonstrated that subjects only retrieve a few events orpieces of knowledge and use this sample to infer frequencies andprobabilities of events. Although the retrieved sample may often be rep'iesentative and the inferred probability judgment fairly accurate, there aremany factors influencing retrievability that do not reflect frequency'Hene, in many situations such cognitive processes will yield incorrectjudgments about frequency. Even though all the specific informationretrieved is accurate, the inferred probability may be seriously in error.Nisbett and Ross (1980) have given a recent comprehensive discussion ofsuch biasing factors in human judgment.


27/67

lf so, whrch?get X and 1...

provide evidence that the subject usedstronger in the third case than in thethe first. The verbalizaton in the firstindependent of any memorythe second and third cases,what information the experimenterto try to infer or guesswill accept, and to generate

for estimating the validity(Newell & Simon, 1972) will oftenadequacy of verbalized information,possibilities for response.may have different efnects upona certain kind of information maythe task are important. Subjects mayso as to be able to give theon subsequent trials.verbalization, subjects are seldomspecific instances of their cognitiveasked to retrospect about theirmany trials or to answer generalall the available informationsubjects have access to anTversky and Kahnemanonly retrieve a few events orsample to infer frequencies andretrieved sample may often be rep-judgment fairly accurate, there arethat do not reflect frequency.processes will yield incorrectthough all the specific informationmay be seriously in error.a recent comprehensive discussion of


Partlcular and General RePortsIf the purpose of retrospective probing were to recover memory traces ofsubjects'processes, the appropriate instruction would be to ask them torecall their specific thought processes during particular trials of the experi-ment. For at least two different reasons, such a procedure is rarely used.First, after a series of trials, a subject's memory for individual cognitiveprocesses will be poor and lacking in detail. Moreover, there is a ten-dency for recurrent cognitive processes gradually to become automatic, sothat fewer or none of the intermediate states of the processes for thelater trials of the experiment are accessible for recall.Second, many experimenters, because they are interested in generalcharacteristics of the thought processes and not in the episodic details ofthe individual trials, probe their subjects with questions of the type,"How did you do these tasks?" Such questions implicitly or explicitly re-quest a general rather than specific interpretation of how the subjectswere performing the tasks.

There are several different ways in which subjects might arrive atdescriptions of their general -procedures, as distinct from reports onspecific behaviors during individual trials. One possibility is that subjectsare aware of the general procedures, or "programs,",they are using, useessentially the same programs on all trials, and can recall and report thesedirectly without reference to the specific behavior they produced.Another pssibility is that subjects remember some parts of theirprocesses during particular trials, and generalize this information into ageneral procedure, which they then report. A different possibility is thatsubjects remember some specific tasks, regenerat+by redoing thern-theprocesses used for these tasks, and use this information to infer thegeneral procedures they may have used. Finally, subjects may draw uponvarious kinds of prior information, such as general knowledge on howone ought to do these tasks, to generate a verbal report describing ageneral procedure or strategy. In this case, the verbal reports may notbear any close relation to the actual cognitive processes (Nisbett & Wil-son, 1977).In areas of applied psychological research where verbal questioninghas a long tradition, subjects are usually asked about specific eventsrather than for general information or conclusions. In the critical incidenttechnique proposed by Flanagan (1954), the subjects were always askedto report their memory for speciflrc events, For example:


28/67

24 Protocol AnalYsis... pilots returning from combat were asked "to think ofsome occasion during combat flying in which you personallyexperienced feelings of acute disorientation or strong vertigo."They were then asked to describe what they "saw, heard, orfett itrat brought on the experience." (Flanagan, 1954, p' 329)

tr"il-il luii;;d, the subjectsIilt, Uut Placed theIn a replicatioOonnell (1963) weiiuuted two artifwhere correct Placeiuessing (the subjecirrt pift hen theYknew the correct ansthe 48.40/o in which tSecond, withO'Connell found thambiguous for theiects who were toldas did the subjects inIn a detailedtrial, DulanY and Osubjects Put the carand O'Connell imPVerplanclt and Oskwere inconsistent wNumerous stuverbalized rules, cand succeeding bSchwartz (1966) adid, the reasons giv1,962 trials. Evenobtained retrosPecresponses to fourfeedback trials eacin more than 90%ApParent InrdequIn a recent extenstive verbal reportthat appears at fr

Interpretive probing, unlike the critical incident technique, cannotbe relied upon to produce data stemming directly from the subjects'ac-tual sequences of thought processes. The probing procedures encourageor even require subjects to speculate and theorize about their processes,rather than leaving the theory-building part of the enterprise to the ex'perimenter. There is no reason to suppose that the subjects themselveswill or can be aware of the limitations of the data they are providing.Moreover, the variety of inference and memory processes that might be r$involved in producing the reports make them extremely dificult to inter'r$pret or to use as behavioral data. 'TWO CHALLENGES TO VERBAL REPORTS

.tIt will be useful, in order to get a perspective on the issues, to use the ,'iabove analysis to examine two published papers that have sometimes ibeen interpreted as providing strong evidence against trusting verbalreports as data from which cognitive processes can be inferred: the ftrst, apaper reporting a study by Verplanck and Oskamp; the second, thereview paper on retrospective verbal reports by Nisbett and wilson. Adiscussion of these papers will show how the information processingmodel we have outlined can help us interpret the findings of experimentson verbalization.Apparent Inadequacies of Concurrent VerblizationIn an often cited study (Verplanck, 1962), Verplanck and Oskampclaimed to have shown that verbalized rules are dissociated from the be-havior they were supposed to control. By having subjects verbalize therules they were following in softing illustrated cards, the experimenterscould reinforce either the verbal rule or the placement of cards (i'e',behavior). To make the contingencies less noticeable, the criterion trialswere followed by additional trials with partial reinforcement. When cor-rect placements were reinforced, the subjects were found to place cards


29/67

combat were asked "to think offlying in which you personallydisorientation or strong vertigo."what they "saw, heard, or(Flanagan, 1954, p.329)the critical incident technique, cannotdirectly from the subjects'ac.The probing procedures encourageand theorize about their processes,part of the enterprise to the ex.suppose that the subjects themselvesof the data they are providing.and memory processes that might bemake them extremely dificult to inter-:,REPORTS ,.]

iii.a perspective on the issues, to use the llpublished papers that have sometimes r,evidence against trusting verbal 'processes can be inferred: the frst, aand Oskamp; the second, thereports by Nisbett and Wilson. A ,show how the information processing ,us interpret the findings of experimentsVerbalization

1962), Verplanck and Oskamprules are dissociated from the be-By having subjects verbalize theillustrated cards, the experimentersrule or the placement of cards (i.e.,less noticeable, the criterion trialswith partial reinforcement. When cor-he subjects were found to place cards


correctly in7l.80/o of the trials; but they stated a correct or correlated rulein only 48,40/o of the trials. When correct statement of the rule was rein,ford, the subjects stated a correct or correlated rule on 92.80/o of thetrials, but placed the cards correctly on only '16.80/o of the trials.In a replication and analysis of this experiment, Dulany andOconnell (1963) were able to show that the above results could be at-ffibuted to two artifacts of the original experiment, First, in the casewhere correct placement was reinforced, by making a correction forguessing (the subjects had a 50-50 chance of placing the card in the cor-rect pile hen they didn't know the rule), we can estimate that subjectsknew the correct answer in 43.60/o of the trials-a percentage very close tothe 48,40/o in which they stated the correct rule.Second, with respect to the reinforcement of rules, Dulany andO'Connell found that the rules defined by Verplanck and Oskamp wereambiguous for the card illustrations they employed. [n fact, naive sub-jects who were told these rules explicitly misplaced the cards as frequentlyas did the subjects in the original experiment.In a detailed analysis of the rules the subjects verbalized on eachtrial, Dulany and O'Connell found that on all but ll of 34,408 trials thesubjects put the card where they said they were going to. Hence, Dulanyand OConnell impeached rather thoroughly the evidence put forth byVerplanclt and Oskamp for believing that the rules subjects verbalizedwere inconsistent with their behaviors.Numerous studies provide positive support for consistency betweenverbalized rules, concepts, and hypotheses and immediately precedingand succeeding behavior, before subjects receive feedback. \JVhenSchwartz (1966) asked subjects their reasons for placing a card as theydid, the reasons given were consistent with the placements on all but 2 of1,962 trials. Even more impressive, Frankel, Levine, and Karpf (1970)obtained retrospective reports from subjects about the basis for theirresponses to four earlier discrimination-learning problems with 30 non-feedback trials each, and found that subjects could provide such reportsin more than 900/o of the sequences of trials.Apparent Inadequacies of Retrospective Reports

In a recent extensive review of studies permitting evaluation of retrospec-tive verbal reports, Nisbett and Wilson 0977) have reported evidencethat appears at first sight to be very damaging to the utility of verbal


30/67

26 Protocol AnalYsisfeports for inferring information processes. since their papgr _haslreceived widespread attention, it is important that we review their find''ings carefully. The authors summarize their main empirical findings thus(tg'11 , p. 233):

We may compathat, according to Nikinds of informationcould report. The incurrent sensations,knows the intermedexactly the kinds ofheld in STM and beUnfortunatelY,fittle data on whatand what informatioverbal rePort' Nisquestons about thenst"ets on memoryprocesses 0977, P.When reinterrogate a mon the stimu

plicit, a Priorstimulus and rln reviewingprofitably raise thamemories of cognprocesses. It is easdence of verbal anactual course and rcessible memory fomation while the Pclusion appears notThe accuracYelicit them and thactual sequence ocussed and obtaineto thoughts (thereports, or requesthoughts were accdescribe in somereportable. Althougfor such informatverbal report Proce

People often cannot report accurately on the effects ofparticulai stimuli on higher order, inference-base$ responses'irrd.;il;"*"times theylannot report on the existence of criti-ct stiun, sometimes cannot report on the existence of their,rrponrrr,'and sometimes cannot even report that an inferen-tial process of any kind has occurred'First, we call attention to the frequent use, in their summary, of the :qualifiers i.often" and "sornetimes." Nisbett and \ililson cite a.large :number of experiments that support their conclusions, but do not inves- 'tigate in detail the conditions uno.t which these conclusions do and donot hold. Moreover, they do not propose a definite model of the cog'nitive processes as a framework for interpreting the findings they survey'Their theoretical interpretations are entirely informal, resting heavily onan undefined distinction between introspective access to "content" and to"process," or, as they alternatively state i, (1977, p' 255)' between ac'css to ';private facts'i and to "rnental processes"' Their summary of thekinds of information to which subjects do have access is this 0977,p. 255): ...wedoindeedhavedirectaccesstoagreatstorehouseof private knowledge ... The individual knows a host of per-,onil hirtorical facti; he knows the focus of his attention atany given point of time; he knows what his curre sensationsare nd has what almost all psychologists and philosopherswould assert to be "knowledge" at least quantitatively superiorto that of observers concerning his emotions, evaluations, andplans. Given that the individual does possess a great deal ofaccurate knowledge ... it becomes less surprising that people*""10 persist in blieving that they have, in addition, direct ac-cess to their own cognitiue processes. The only mystery iswhy people are so poor at telling the difference betweenpriiat facts that ..n , known with near certainty and mentalpto.".r"t to which there may be no access at all'Nisbett and lJVilson also observe that subjects "are often capable ofdescribing intermediate results of a series of mental operations $977,p. zss),,1i."., th.t they hold in sTM.and can access the symbols that areinputs and outputs to such operations)'


31/67

processes. Since their paper hasis important thar we review if," n"i.rtheir main empirical fi"di;;th;*

use, in their.summary, of the $Nisbet and Wilson ;i"'i lrheir conclurion., uui;'il#:: fwhich rhese concluron, Jo u'

accurately on the effects oforder, inference-barrl r;;;;:report on the existnce f ,rt_report on the existence of th;l;even report that an infern-

propose a definite model "f ,lre;"* iinterpreting the nnOings the;;;r;'entirely informal, resting nuulv onaccess to ,toneniJ"io

it, (1977, p.255), br*.;;-.;:processes." Their summ.rt;i th,do have access is this (lgi.


We may compare this list of "private facts" and intermediate resultsthat, according to Nisbett and Wilson, are accessible to subjects with thekinds of information that our processing model would imply that subjectscould report. The individual knows, they say, his focus of attention, hiscurrent sensations, his emotions, his evaluations, and his plans. Heknows the intermediate results of his mental operations. But these areexactly the kinds of information that, according to our model, would beheld in STM and be available for verbal reports.Unfortunately, the studies reviewed by Nisbett and Wilson providettle data on what information is heeded during the thought processes,and what information is accessible from STM and LTM at the time of theverbal report. Nisbett and Wilson find that the subjects, when askedquestions about their cognitive processes, frequently do not base theiranswers on memory for specific events at all, but "theorize" about theirprocesses 0977 , p. 233).When reporting on the effects of stimuli, people may notinterrogate a memory of the cognitive processes that operatedon the stimuli; instead, they may base their reports on im-

plicit, a priori theories about the causal connection betweenstimulus and response.In reviewing the studies cited by Nisbett and Wilson, we canprofitably raise the question of why and when subjects do not consult theirmemories of cognitive processes in answering questions about thoseprocesses. It is easy to draw the erroneous conclusion that this indepen-dence of verbal answers to questions about cognitive processes from theactual course and results of those processes implies a generallack of ac-cessible memory for such processes, or even an unawareness of the infor-mation while the process was actually going on. But this sweeping con-clusion appears not to be justified.The accuracy of verbal reports depends on the procedures used toelicit them and the relation between the requested information and theactual sequence of heeded information. Invalid reports, like those dis-cussed and obtained by Nisbett and Wilson, may be due to lack of accessto thoughts (their claim), inadequate procedures for eliciting verbalreports, or requesting information that could not be provided even ifthoughts were accessible. In a subsequent chapter (Chapter 3) we willdescribe 'in some detail what information will be heeded and hencereportable. Although some studies cited by Nisbett and Wilson did probefor such information, we will focus here on the deviations between theverbal report procedures used in many of the studies cited by Nisbett and

access to a grear storehouseknowJ a host "i;;;:the focus of his uttrniloitwhat his currenr sensationsand pfrirprrleast quantitatively ;;i;;ltr. emoons, evaluarionJ, anddos possess a great deal oftess surprising rhat peoplehave, in addition, Oi..ic_The only mystery isthe difference betwlennear certainty and mentalaccess at all.that subjects ..are often capable ofof mental operations eg77,can access the symbols that are


32/67

28 Protocol AnalYsisWilson and the procedures that, according to our model' wouldvalid retrospective reports of cognitive processes' ';:r'---First, many of the verbaireports they discuss could be generatedwithout accessing memofy of the corresponding cognitive processs. tnsome of these studies, the questions presented to subjects ::1t::T

Finally, in somewere asked to reportIit. ut*.. to theu*0, and informatg"ititt Processeuttto bY Probes^rrndirected requests lte studies in the Nisio oUtain from the Proi*ttut. For examnnot ans\ryer fromierenttY from subjecdid not include sueriveO, if at all, oprovided bY differe,tuOitt the subjectsoerimental conditioprobing for hypothecognitive Processes'other studies, subjeor generalize the Proroartttt used on eSeveral articleNisbett and Wilsonparticular interestreports in betweenstdies have showare able to Proviwright, 1979; WrigIn sum, we

subjects simply weexPeriments' Instawareness (concuconsiderable insighstudies discussedwith Nisbett and Wthe subjects do nmemory for the cbett and Wilson Phis own behavior

to retrieving information from memory' r--,^- -^ ISecond, several .*tttt of the vrbal report procedures reviewed by Iigconor sevglar aPeer vr uv --:-- .' ,- '',rr^NisbettandWilson*u"tt."relevantthoughts|essaccessible,lnmostof the studies reviewed,-the time lag between task and probe-wassuf',,

siderable background iirformation from which answers could be generated"without consulting tn; memories. with questions like, "I noticed that,you took .or, .nort than'average' Why do you suppose you did?r'(Nisuett & wilson, lsll, p.23?) Iiis not even clear to us, nor

Documents

Ericsson verbal protocol