Houlette Linguistics

8/12/2019 Houlette Linguistics

1/8

LINGUISTICS, EMPIRICAL RESEARCH,ND EVALUATING COMPOSITION

Forrest Houlette

t seems that David Hume's mother once said LittleDavie means well, but he's not very bright. That's okay, saidthe philosophy professor who told me the story, because shewas wrong on both counts. l But very often we hold Davie'smother's sentiments toward empiricists because we feel thatthey needlessly attack and destroy some of our favorite assumptions about writing, the universe, and everything. Michael Holzman's recent argument that composition research must avoidscientism and use sodal scientific methodologies in the serviceof a humanistic pedagogy is a case in point. Holzman accusesempirical researchers, particularly those involved with sentencecombining, of pursuing mathematical models and statisticalanalyses for the sake of securing a validity for their researchwhich it otherwise would not have had, of wearing those gorgeous cloaks over a poor reality. He is willing to admit that asocial scientific methodology eventually may be valuable in literacy research, but he jealously guards the role of nonempiricalstudy, stating that empirical, social scientific methodologies willnot necessarily be superior to humanistic modes of research. 2The tone of the superiority of the nonempirical humanistic overthe empirical social scientific is evident, and it aligns Holz-manwith little Davie's mother. The problem, though, is that theempirical and the nonempirical do not conflict. What thetypical humanistic dismissal of scientism does not concede isthat we need the empirical point of view precisely because itattacks our assumptions. Empiricists keep us honest in important ways. We must therefore always keep in mind who they areand what they can do for us.

JOURNAL OF ADVANCED COMPOSmON, Volume V (1984). Copyright1988.


2/8

108 Journal o Advanced Composition

At first glance, empiricists may seem to be people whotrust only what they see, people like Bishop Berkeley, whodenied the existence of the physical world apart from the perceiving mind.3 Since Heisenberg and his uncertainty principleentered the scene, however, they do not even trust what they dosee.4 They are more likely to believe in what they can measure,with the caveat that how they measure influences the way theyunderstand the measurements. Empirical researchers go abouttesting assumptions by predicting the measurements they oughtto get based on those assumptions and comparing those measurements to the measurements they actually get. They use statisticsto tell how large a difference between what they wanted andwhat they got means that the assumption is invalid.Empiricists are not by nature antihumanists. The empirical spirit grew out of the Renaissance and Reformation alongwith Humanism. The critical difference between the two wasthat the Humanists favored classical texts as having the power toexplain the physical world while the empirical spirit favoredobservation of nature.s Both traditions continued in philosophy,the Humanists most probably tracing their roots through Descartes cogito ergo sum the empiricists through Berkeley s esseest perdpi. The conflict evident in Holzman s argument easilyreduces to the root conflict which spawned the differentiation ofempiricism from Humanism. The question is epistemological:whether we shall go to learned texts and apply the techniques ofrationalism to deduce what is the case, or whether we shallobserve nature and induce what is the case. The answer to thisquestion would appear to be necessarily a choice between oneoption or the other. But this conflict is really trivial. Anyempirical study must rely on the products of intuitive, introspective, or rationalistic types of reasoning. Hypotheses comefrom observation, critical observation, yet these observations arenot rigorously controlled. Hypotheses are guesses which reflectall of the personal biases of the observer biases based on intuitive, introspective, and rationalistic modes of reasoning. Theyare the result of the observer projecting these biases on theworld. The controlled observation of the experiment is merely atest to see whether these hunches can be considered correct. Anyhumanistic study must rely to a certain extent on empirical observation of the world. Descartes, for instance, discusses severalmetaphors based on his observation of the world, and thesemetaphors shape and inform the premises of his deductive argu-


3/8

Forrest Houlette 109

ments.6 In rationalistic methods, however, there is no rigorouscontrol on the observation, the derived metaphor, or its inter-pretation. The rationalistic and interpretive methodologies ofthe humanities therefore complement the experimental method-ologies of empiricism. Humanistic methodologies yield the stuffof hypotheses; empirically proven hypotheses provide furthergrounds for humanistic interpretation. The two sets of disci-plines must rely on each other by nature.Theoretical inquiry in the humanistic fields of composi-tion study is a method which proceeds largely by introspectionnd argument by example. What we believe about writing ndhow to teach it depends on how we perceive ourselves as writersnd what we accept as examples of good writing. To teach thecomposing process, we must first believe we understand how weourselves compose. We must also understand from observationhow we believe our process might differ from another writer'sprocess. We will teach only wh t coincides with our understand-ing of process. To teach style, we must first understand what wethink is good style. We will rely on examples from literature forthis knowledge. To teach organization, coherence, or unity, wemust first understand these concepts. Again, we will rely onexamples to establish this knowledge.Empiricism, when applied to composition research, mustinherently take a linguistic or behavioral approach to discoursebecause of the emphasis on measurement. In order to measuresome feature of a text or its impact on a reader, we must be ableto convert aspects of the text or of the response to numbers. f

we wish to measure a text, we must count grammatical features,coherence markers, propositions, features of context, or perhapsfeatures of suprasentential structures, for all of which we mustdepend on linguistic descriptions of the discourse. To describerhetorical or cognitive features of the text, we must still countthem in terms of the linguistic features which evidence them. fwe wish to make measurable statements about the writing of thetext, we must observe, record, claSSify and count the writer'sbehaviors during the production of the text. f we intend tomeasure a reader's response, we must depend on some sort of asemantic differential scale. Linguistics and psychology are there-fore the pure sciences which an empirical composition re-searcher applies to texts in order to verify insights suggested bythe less empirical disciplines of rhetoric and literary study.The role for the humanist within composition research is


4/8


5/8

Forrest Houlette 111

writing assignments were inherently more valid measures ofwriting skill than any external criterion known. When henoticed that Hunt s factors and a few of his own devising did notcorrelate well with such grades, his reaction was to reject theexternal criteria as valid measures of writing skill and to question the validity of the theory which produced them. Faigley sreaction, however, was a function of his faith in the assumptionthat reliably assigned holistic scores were indeed valid measuresof writing skill, a humanistic assumption since its relies on anepistemology based on the reading of texts and the judgments ofinformed readers. There is a counter argument, though, anotherway of posing the problem, which Faigley did not consider, probably because of his faith in his assumption. Instead of questioning the ability of the variables to correlate with grades, he couldhave questioned the ability of grades to correlate with variables.What if the external criteria were indeed the more valid measures? This is an empirical question since it relies on an epistemology which values external verification.Recently I undertook to investigate this alternative way ofposing the question. What I found was not terribly surprising.First, almost no one had considered the question. Of the fewstudies available, only one found a strong correlation betweengrades and independent measures)l The outlook for successwith studies of the empirical validity of holistic scores was therefore bleak. Second, none of the research was overly concernedwith the statistical significance of the correlations, whether theymight have been produced by a chance interaction of actuallyunrelated factors. Procedures were just lax enough to inflateerror levels unacceptably. Third, a growing body of psychologicalliterature indicated that human beings do not necessarily makevalid decisions, a fact which calls into question Faigley s assumption of the validity of holistic scores 12 Grading is a process ofmaking decisions about which of several categories to consign aseries of writing samples to. f graders are not necessarily goodjudges, their grades do not possess the inherent validity Faigleybelieved them to have. Because of these three factors, I decidedto design an exploratory study to see whether independentcriteria might be found which would correlate well with grades.The study employed three sets of reliably graded papersand seven variables, Hunt s clause to sentence factors and twovariables derived from the theory of given and new information, content words per t-unit and the percentage of content


6/8

112 Journal of dvanced Composition

words marked as given information.13 What it revealed is thatunder some circumstances empirical validity is possible. Fortwo sets of papers, the correlations would not be distinguishedfrom zero. For the other set, however, a combination of twovariables, percentage of content words given nd words per sentence, explained twenty-three percent of the variance in thegrades.14 As studies like this one go, that is a fairly high percentage. The three sets of papers were collected under a mixed bagof conditions involving both basic nd intermediate writers,primary trait nd general impression scoring, nd contextual ndnoncontextual assignments. The intent was to try to catch agroup of conditions that could produce validity. In this sense,the study was successful. The results favored intermediate overbasic writers, primary trait scoring over general impression scoring, nd contextual assignments over noncontextual assignments. What this study tells researchers in composition who d-here to humanistic methodologies is th t the assumption of thevalidity of reliably assigned holistic scores is a dangerous assumption. t would seem that such scores can t some times be morevalid than t other times. What this study tells researchers whofollow the empirical tradition is that they need more fully toexplore the relationships between holistic scores nd externalcriteria. t may be th t such research could verify a set of conditions under which the most valid grading could be done or thatsuch research could identify the theories most useful in constructing external measures of writing ability. The two possibilities coincide with the two ways of viewing the problem mentioned above.) The study implies research programs for researchers of both persuasions. The questions nonempirical theoristsmust explore are when nd how the validity discovered mightbe produced or improved upon. The question th t empiricaltheorists must address is whether the nonempirical answerswork out well in practice.

f we are going to find the answers to such questions, weneed to rely on both types of researchers nd we need to be crosstraining researchers in both lines of inquiry. We need to be ableto communicate with one another, nd interdisciplinary training, since it promotes understanding, is the most effectivemeans of facilitating communication. These are the reasonswhy the linguistic, the psychological, nd the empirical havebeen adopted by researchers in composition nd teachers of com-


7/8

Forrest Houlette 3

position theory. What empiricists do enables work by their nonempirical colleagues which in tum enables their own work. Therelationship is not the antithetical one Holzman implies.Rather, it is very like mine with my checkbook. I had gonealong several months just believing that I had not made a mistake, a nonempirical stance in the extreme. But I got out myvery empirical calculator recently. I found two hundred dollars.Ball State UniversityMuncie, Indiana

Notes

1 G. Stanley Kane told me the story while I was at Miami University. Ihave not been able to trackdown its authenticity.2 Michael Holzman, ''Scientism and Sentence Combining, College Com-position and Communication voL 34, no. 1 Feb. 1983), pp. 79, 74, and 78, respectively.3 For an introduction to Berkeley's ideas, see W. T. Jones, History ofWestern Philosophy 2nd ed., voL llI Hobbes to Hume (New York: Harcourt,Brace, andWorld, 1969), pp. 280-95.4 Werner Heisenberg provides a relatively understandable explanationof his uncertainty principle and related issues in Physics and Beyond (NewYork:HarperandRow, 1972), pp. 58-81.5 See Jones, Hobbes to Hume p. 68.6 For an introduction to Descartes' method, see Jones, Hobbes to Humepp.154-91.7 For an excellent discussion of the concept of validity, see David P.Harris, Testing English as a Second Language (New York: McGraw-Hill, 1969),pp.18-21.8 See Frank O'Hare, Sentence Combining: Improving Student WritingWithout Formal Grammar Instruction (Urbana, IlL: NCI'E, 1973), p. 67, KelloggW. Hunt Grammatical Structures Written at Three Grade Levels (Champaign,Ill: NCI'E, 1965), and Walter D. Leban, Language Development: KindergartenThrough Twelfth Grade (Urbana,Ill.: NCI'E,1976).9 See Paul B. Diederich, Measuring Growth in English (Urbana, I l lNCI'E,1974).10 Lester Faigley, ''Names in Search of a Concept: Maturity, Fluency,Complexity, and Growth in Written Syntax, College Composition and Commun-ication ,31, no. 3 (Oct. 1980),291-300.


8/8

114 Journal of Advanced Composition

11 Mary Lou Howerton, The Relationship between Quantitative ZndQw.zlifative Mi Ilsure5 of Writing SkiUs, ED 137 416 See also Linda Brodkeyand Rodney W. Young, Sensible, Interesting, Orgtmized, RJreturiall Procedurefur the Grading of Student Essays, E 173 005, and Sean A Walmsley and PeterMosenthal, Psycholinguistic BIlses fur Holistic Judgements of Children's Writ-ten Discourse, ED 177

12 See Leon Rappoport and David A Summers, ed., Humtl7l Judgment ZndSocilZl InterRCtion (New York: Holt, Rinehart, and Winston, 1973) for severalreview articles and bibliographies.

13 Content words per t-unit is a general measure of the number of propositions present in discourse. Percentage of content words marked as given information is a measure of the information shared by writer and reader. Given information is defined within the theory as information shared by writer andreader in consciousness. New information is information not so shared, that is,information which needs to be communicated from writer to reader in order to beshared. For further explanation of the variables, see Forrest Houlette, ARegression Model of the Grading Process Employing lriables Drawn from theTheory of Given Rnd New InfurmlZtion, Doctoral Dissertation, University ofLouisville. For further explanation of the theory, see Herbert H Clark andSusan E Haviland, Comprehension and the Given-New Contract, in Dis-course Processes: AdVtl1lCes in Resetlrch Rnd Theory ed. Roy O. Freedle, voL I:Discourse Production and Comprehension (Norwood, N J : Ablex Publishing,1977), pp. 1-40.14R= 492

Documents

Houlette Linguistics