Research Methods of Research Methods of Applied Linguistics and Applied Linguistics and

StatisticsStatistics

By Qin XiaoqingBy Qin Xiaoqing((Huazhong University of Science and Huazhong University of Science and

TechnologyTechnology))

22

Lecturer: Lecturer: 秦晓晴秦晓晴Email: Email: qxq@mail.hust.edu.cn

Public email: Public email: englishgrad@126.com

Password: 123456Password: 123456

33

An Introduction to this CourseAn Introduction to this Course

Definitions & classifications of research, Definitions & classifications of research, and research procedures and research procedures

The concept of variables The concept of variables

Quantitative research design Quantitative research design

Questionnaire surveyQuestionnaire survey

Item analysis,validity & reliabilityItem analysis,validity & reliability

Data exploringData exploring

44

Descriptive statisticsDescriptive statistics

Inferential statisticsInferential statistics

Writing a research proposalWriting a research proposal

Reporting a researchReporting a research

Research evaluationResearch evaluation

55

Nature of This CourseNature of This Course

To give lecturesTo give lectures

To hold discussions and To hold discussions and presentationspresentations

To do self-studyTo do self-study

To administer a test To administer a test

66

Course books and references (1)Course books and references (1)

Mackey, A., & Gass, S. M. (2005). Mackey, A., & Gass, S. M. (2005). Second Second Language Research : Methodology and Language Research : Methodology and DesignDesign. Mahwah, NJ: Lawrence Erlbaum.. Mahwah, NJ: Lawrence Erlbaum.

Brown, J. D. (2001). Brown, J. D. (2001). Using Surveys in Using Surveys in Language ProgramsLanguage Programs. Cambridge: . Cambridge: Cambridge University Press.Cambridge University Press.

Dörnyei, Z. (2003). Dörnyei, Z. (2003). Questionnaires in Questionnaires in Second Language Research: Construction, Second Language Research: Construction, Administration and ProcessingAdministration and Processing. Mahwah, . Mahwah, N.J.: Lawrence Erlbaum Associates.N.J.: Lawrence Erlbaum Associates.

77

Course books and references Course books and references (2)(2)

Hatch, E. M., & Lazaraton, A. (1991). Hatch, E. M., & Lazaraton, A. (1991). The The Research Manual: Design and Statistics for Research Manual: Design and Statistics for Applied LinguisticsApplied Linguistics. New York, NY: Newbury . New York, NY: Newbury House Publishers.House Publishers.Seliger, H. W. & E. Shohamy, 1999. Seliger, H. W. & E. Shohamy, 1999. Second Second Language Research Methods. Language Research Methods. Shanghai:Shanghai: 上海外上海外语教育出版社语教育出版社 ..刘润清，刘润清， 19991999 ，，《《外语教学中的科研方法外语教学中的科研方法》》。。北京：外语教学与研究出版社。北京：外语教学与研究出版社。秦晓晴，秦晓晴， 20032003 ，，《《外语教学研究中的定量数据外语教学研究中的定量数据分析分析》》。武汉：华中科技大学出版社。。武汉：华中科技大学出版社。文秋芳，文秋芳， 2004,2004, 《《应用语言学研究方法与论文写应用语言学研究方法与论文写作作》》。北京：外语教学与研究出版社。。北京：外语教学与研究出版社。

88

Definition of ResearchDefinition of Research

1.Dictionary definitions1.Dictionary definitions

(1) According to Longman Dictionary, (1) According to Longman Dictionary, research means advanced study of a research means advanced study of a subject, so as to learn new facts on subject, so as to learn new facts on scientific laws. scientific laws.

(2) Collin’s Dictionary says research is a (2) Collin’s Dictionary says research is a detailed study of a subject or an aspect of detailed study of a subject or an aspect of a subject.a subject.

99

2. Formal definition of research2. Formal definition of research

Research is a systematic approach to Research is a systematic approach to finding answers to questions.finding answers to questions.

question—appropriatequestion—appropriate

approach—systematicapproach—systematic

answer—validanswer—valid

1010

Purposes of researchPurposes of research

(a) increasing knowledge within the (a) increasing knowledge within the discipline of applied linguistics, and discipline of applied linguistics, and

(b) increasing knowledge within oneself to (b) increasing knowledge within oneself to understand new developments within the understand new developments within the discipline. discipline.

1111

How to increase knowledgeHow to increase knowledge

supports the supports the theoretical basis theoretical basis of the of the disciplinediscipline

providing evidence for the providing evidence for the efficacy efficacy of a of a curriculum, EFL teaching and learning, and curriculum, EFL teaching and learning, and so on.so on.

Developing Research Tools.Developing Research Tools.

For most students, the ability to understand For most students, the ability to understand research in applied linguistics may be more research in applied linguistics may be more important than making a research important than making a research contribution to the profession. contribution to the profession.

1212

Definition of theoryDefinition of theoryDictionary: A Dictionary: A theorytheory is a formal idea or set is a formal idea or set

of ideas that is intended to explain of ideas that is intended to explain something. something.

Kerlinger (1986: 9): A theory "is a set of Kerlinger (1986: 9): A theory "is a set of interrelated constructs (concepts), interrelated constructs (concepts), definitions, and propositions that present a definitions, and propositions that present a systematic view of phenomena by systematic view of phenomena by specifying relations among variables, with specifying relations among variables, with the purpose of explaining and predicting the purpose of explaining and predicting the phenomena" .the phenomena" .

1313

Types of research Types of research

Adapted from Brown Adapted from Brown 2001:12001:1

1414

Characteristics of statstical researchCharacteristics of statstical research

Systematic (systematically structured with Systematic (systematically structured with definite procedural rules)definite procedural rules)

Logical (based on a step-by-step logical Logical (based on a step-by-step logical pattern)pattern)

Tangible (based tangible, quantifiable Tangible (based tangible, quantifiable information, ie data)information, ie data)

Replicable (able to repeat the research)Replicable (able to repeat the research)

Reductive (help form patterns in the Reductive (help form patterns in the seeming confusion of facts)seeming confusion of facts)

(J. D. Brown 1988)(J. D. Brown 1988)

1515

Steps of quantitative research Steps of quantitative research

(1) Ask or form appropriate questions(1) Ask or form appropriate questions

(2) Review the literature(2) Review the literature

(3) State hypothesis(3) State hypothesis

null (Hnull (H00) alternative (H) alternative (H11): positive, ): positive,

negativenegative

(4) Select an approach and do the (4) Select an approach and do the researchresearch

(5) Analyze the data(5) Analyze the data

(6) Report results(6) Report results

1616

Sources of Research QuestionsSources of Research Questions

Step 1: Formulating a general research Step 1: Formulating a general research questionquestion

Interest and personal experienceInterest and personal experience

Journals of our fieldJournals of our field

Supervising facultySupervising faculty

Sources outside second languageSources outside second language

1717

Step 2: Narrowing down or Step 2: Narrowing down or focusing the questionfocusing the question

Importance (generate new information or Importance (generate new information or confirm old information in new ways) confirm old information in new ways)

FeasibilityFeasibility

List the possible factors comprising the List the possible factors comprising the general questiongeneral question

1818

Eg. How well do L2 learners Eg. How well do L2 learners perform speech act?perform speech act?

(Broad question)(Broad question)Which speech acts do you want to Which speech acts do you want to investigate?investigate?What types of L2 learners are you going to What types of L2 learners are you going to choose?choose?In what kinds of situations should the In what kinds of situations should the speech events be investigated?speech events be investigated?Is the research meant to support test Is the research meant to support test development? Material development? development? Material development? Theory developmentTheory development

1919

Narrowed QuestionNarrowed Question

What is the Chinese EFL learners’ ability to What is the Chinese EFL learners’ ability to recognize complaint behavior appropriate in recognize complaint behavior appropriate in an academic university setting?an academic university setting?

Total performance –recognition of one Total performance –recognition of one portion of the totalportion of the total

L2 learners – Chinese EFL learnersL2 learners – Chinese EFL learners

Speech acts – complaint (subcategory)Speech acts – complaint (subcategory)

The overall setting – university settingThe overall setting – university setting

2020

Activity 1Activity 1

Formulate a research question you are Formulate a research question you are interested ininterested in

2121

Activity 2Activity 2

Evaluate the following question:Evaluate the following question:

What are the characteristics of the good What are the characteristics of the good language learner?language learner?

2222

From theoretical definition to From theoretical definition to operational definitionoperational definition

Any terms must be operationally definedAny terms must be operationally defined

An operational definition is a clear An operational definition is a clear statement of how you judged or identified statement of how you judged or identified a term in your research a term in your research

Good operational definitions can often be Good operational definitions can often be drawn from the existing literaturedrawn from the existing literature

2323

Criterion for a good research Criterion for a good research questionquestion

The question shouldThe question should

Interest usInterest us

Promise new information or confirm old Promise new information or confirm old information in new ways (contribution)information in new ways (contribution)

Have reasonable scope (feasibility)Have reasonable scope (feasibility)

Have key terms that are clearly defined Have key terms that are clearly defined and operationalizedand operationalized

2424

Factors influencing feasibilityFactors influencing feasibility

The amount of time and cost neededThe amount of time and cost needed

Researcher’s prerequisite knowledgeResearcher’s prerequisite knowledge

Consistent definition of concepts and Consistent definition of concepts and termsterms

Access to dataAccess to data

2525

The purposes of research The purposes of research questionsquestions

DescriptiveDescriptive

ExploratoryExploratory

Explanatory Explanatory

2626

Step 3: Review the literature on the Step 3: Review the literature on the topic as completely as possibletopic as completely as possible

Which questions have been answered Which questions have been answered satisfactorily?satisfactorily?

Which seem trivial?Which seem trivial?

Which are too complex to tackle at the Which are too complex to tackle at the moment?moment?

Which procedures have been used in Which procedures have been used in answering the questions?answering the questions?

What results have been obtained?What results have been obtained?

2727

Step 4: Formulating the research Step 4: Formulating the research question or the hypothesisquestion or the hypothesis

A hypothesis is a statement of possible A hypothesis is a statement of possible outcome of research.outcome of research.

Null hypothesis and alternative Null hypothesis and alternative hypotheseshypotheses

2828

HH11 andand HH22

There is a positive relationship between There is a positive relationship between similarity of relative clause structures in the similarity of relative clause structures in the learners’ L1 and L2 and their performance on learners’ L1 and L2 and their performance on L2 relative clauses.L2 relative clauses.

There is a negative relationship between There is a negative relationship between similarity of relative clause structures in the similarity of relative clause structures in the learners’ L1 and L2 and their performance on learners’ L1 and L2 and their performance on L2 relative clauses.L2 relative clauses.

2929

HHoo

There is neither positive nor negative There is neither positive nor negative relationship between similarity of relative relationship between similarity of relative clause structures in the learners’ L1 and clause structures in the learners’ L1 and L2 and their performance on L2 relative L2 and their performance on L2 relative clauses.clauses.In most research reports, the null In most research reports, the null hypothesis, even if not formally stated, is hypothesis, even if not formally stated, is tested rather than a directional alternative tested rather than a directional alternative hypothesis. Why?hypothesis. Why?

3030

Activity 3Activity 31.1. Research topic: relationship of language Research topic: relationship of language

proficiency and spelling test scoresproficiency and spelling test scores

2.2. How to state and interpret the null and How to state and interpret the null and alternative directional hypothesesalternative directional hypotheses

Null hypothesis:Null hypothesis:

Alternative hypothesis:Alternative hypothesis:

Directional, positive hypothesis:Directional, positive hypothesis:

Directional, negative hypothesis:Directional, negative hypothesis:

3131

Null hypothesis:Null hypothesis: There is no relationship There is no relationship between l. proficiency and spelling between l. proficiency and spelling scores.scores.

The spelling scores will not vary in The spelling scores will not vary in relation to the l. proficiency of the Ss.relation to the l. proficiency of the Ss.

Interpretation of hypothesesInterpretation of hypotheses

3232

Alternative hypothesisAlternative hypothesis: There is : There is relationship …relationship …

The spelling scores will vary in The spelling scores will vary in relation to the l. proficiency of the Ss.relation to the l. proficiency of the Ss.

3333

Directional, positive hypothesisDirectional, positive hypothesis: There is a : There is a positive relationship between l. proficiency and positive relationship between l. proficiency and spelling scores.spelling scores.

Students at higher proficiency levels will obtain Students at higher proficiency levels will obtain higher spelling scores. Students at lower higher spelling scores. Students at lower proficiency levels will obtain lower spelling proficiency levels will obtain lower spelling scores. scores.

3434

Directional, negative hypothesisDirectional, negative hypothesis: There is : There is a negative relationship between l. a negative relationship between l. proficiency and spelling scores.proficiency and spelling scores.

Students at lower proficiency levels will Students at lower proficiency levels will obtain higher spelling scores while those obtain higher spelling scores while those with higher levels of l. proficiency will with higher levels of l. proficiency will score lower.score lower.

3535

Activity 4Activity 41.1. Research topic: gender differences in spelling Research topic: gender differences in spelling

test scorestest scores

2.2. state a separate null hypothesis for the effect of state a separate null hypothesis for the effect of gender on spelling scores (ignore proficiency)gender on spelling scores (ignore proficiency)

3.3. State all the interaction hypotheses (adding in State all the interaction hypotheses (adding in proficiency and L1 membership)proficiency and L1 membership)

HHo o for gender: for gender:

HHo o for gender and L1:for gender and L1:

HHo o for gender and language proficiency:for gender and language proficiency:

HHo o for gender, L1, and language proficiency:for gender, L1, and language proficiency:

3636

InterpretationInterpretation

HHo o for genderfor gender: There is no effect of gender on : There is no effect of gender on spelling scores.spelling scores.

HHo o for gender and L1for gender and L1: There is no interaction : There is no interaction of gender and L1 membership on spelling of gender and L1 membership on spelling scores. scores.

3737

HHo o for gender and language proficiencyfor gender and language proficiency: :

There is no interaction of gender and There is no interaction of gender and language proficiency on spelling scores. language proficiency on spelling scores.

HHo o for gender, L1, and language for gender, L1, and language

proficiencyproficiency: There is no interaction of : There is no interaction of gender, language background, and gender, language background, and language proficiency on spelling scores. language proficiency on spelling scores.

3838

HHo o for genderfor gender rejectedrejected: males and females : males and females

would show different spelling scores.would show different spelling scores.

HHo o for gender and L1 rejectedfor gender and L1 rejected: there might be a : there might be a

pattern where for females the scores wouldn’t pattern where for females the scores wouldn’t change across the L1 groups while for men change across the L1 groups while for men they would. they would.

3939

HHo o for gender and language proficiency for gender and language proficiency rejectedrejected: there might be a pattern where : there might be a pattern where for females the scores wouldn’t change for females the scores wouldn’t change across the proficiency levels while for men across the proficiency levels while for men they would.they would.

HHo o for gender, L1, and language for gender, L1, and language proficiency rejectedproficiency rejected: it might mean, eg, : it might mean, eg, that females of certain L1 groups who are that females of certain L1 groups who are at certain proficiency levels perform at certain proficiency levels perform differently than others. differently than others.

4040

Answering the question: Internal Answering the question: Internal and external validityand external validity

Internal validityInternal validity is the extent to which the is the extent to which the outcome is a function of the factor you outcome is a function of the factor you have selected rather than other factors have selected rather than other factors you haven’t controlled. you haven’t controlled.

External validityExternal validity is the extent that the is the extent that the outcome would apply to other similar outcome would apply to other similar situations in the real world. situations in the real world.

4141

Factors influencing the internal Factors influencing the internal validityvalidity

Subject selection (size variability, sample Subject selection (size variability, sample size)size)Maturation (developmental changes in Maturation (developmental changes in subjects) subjects) History (no control of input amount beyond History (no control of input amount beyond the experiment)the experiment)Task direction (no clear instructions)Task direction (no clear instructions)AttritionAttritionTest effect (the influence of pretest on Test effect (the influence of pretest on posttest)posttest)

4242

Subject selectionSubject selection

In order for research to be generalizable, In order for research to be generalizable, the sample used should be representative the sample used should be representative of the general population. Methods used to of the general population. Methods used to achieve this representativeness include achieve this representativeness include both both random samplingrandom sampling and the sample and the sample size large enough.size large enough.

4343

Example of subject selectionExample of subject selection

2 groups of learners have been randomly 2 groups of learners have been randomly assigned. Each group contains 10 subjects. assigned. Each group contains 10 subjects. After the instructional treatment, a test is given After the instructional treatment, a test is given to both groups. The average score for group A to both groups. The average score for group A is 9.3 while 11.3 for group B. The standard is 9.3 while 11.3 for group B. The standard deviations for both are 4.15 and 1.56 deviations for both are 4.15 and 1.56 respectively. Can we say the method used respectively. Can we say the method used with group B is superior to that with group A? with group B is superior to that with group A? Why or why not?Why or why not?

4444

With this difference between the groups With this difference between the groups for the SD, we would have to question for the SD, we would have to question whether test differences might be due to whether test differences might be due to some other, unidentified factor for the some other, unidentified factor for the score seem to indicate that the 2 groups score seem to indicate that the 2 groups are drawn from different populations.are drawn from different populations.

The problem for this research is that the The problem for this research is that the subjects are not from a homogeneous subjects are not from a homogeneous population.population.

4545

Maturation Maturation

Maturation refers to the general Maturation refers to the general developmental changes in subjects during developmental changes in subjects during the course of the research esp. in the course of the research esp. in longitudinal studies. The differences may longitudinal studies. The differences may be caused by the treatment , the be caused by the treatment , the maturation in cognitive development and maturation in cognitive development and short-term memory capability, or a short-term memory capability, or a combination of both. combination of both.

4646

HistoryHistory

History refers to the possible negative History refers to the possible negative effects of the passage of time on the effects of the passage of time on the study. For example, learners may have study. For example, learners may have more access to sources of input or use more access to sources of input or use English more outside of class. In this case, English more outside of class. In this case, we can’t say the difference in performance we can’t say the difference in performance is caused by the experimental treatment.is caused by the experimental treatment.

4747

Task directionsTask directions

When the instructions are clear, the results When the instructions are clear, the results may be valid. For example, when a may be valid. For example, when a researcher used an instrument of English, researcher used an instrument of English, the subjects may not understand the the subjects may not understand the instruments accurately. instruments accurately.

4848

AttritionAttrition

By attribution we mean that the By attribution we mean that the composition of the population studied may composition of the population studied may change the longer the study continues. change the longer the study continues. Subjects may lose interest when data Subjects may lose interest when data collection is made. collection is made.

4949

Test effectTest effect

If a research begins with a pretest, that If a research begins with a pretest, that test can affect performance during the test can affect performance during the treatment and on future tests. The test treatment and on future tests. The test alerts subjects as to what teachers expect alerts subjects as to what teachers expect them to learn. Then they may pay much them to learn. Then they may pay much attention to it during the course of the attention to it during the course of the research. research.

5050

external validityexternal validity

If a study has no internal validity, then it If a study has no internal validity, then it cannot have external validity. We cannot cannot have external validity. We cannot generalize from the data.generalize from the data.

When generalizing from the data set, we When generalizing from the data set, we are both concerned with IV but EV as well, are both concerned with IV but EV as well, namely how representative the data are namely how representative the data are for the group(s) to which we hope to for the group(s) to which we hope to generalize. generalize.

5151

Factors influencing the external Factors influencing the external validityvalidity

Population characteristicsPopulation characteristics

The effect of the research environmentThe effect of the research environment

Research effectsResearch effects

Data collection methodologyData collection methodology

5252

Population characteristicsPopulation characteristics

Research carried out with university Research carried out with university students may not be valid when applied to students may not be valid when applied to children. children.

Results from a study of English majors Results from a study of English majors may not be applicable to non-English may not be applicable to non-English majors.majors.

5353

The effect of the research The effect of the research environmentenvironment

Second language learners may become Second language learners may become more motivated simply because they are more motivated simply because they are told that they are participating in a study told that they are participating in a study that will help the researcher understand that will help the researcher understand the process of language learning. This is the process of language learning. This is called the “Hawthorne effect’. called the “Hawthorne effect’.

5454

Researcher effectResearcher effect

The researcher may unintentionally The researcher may unintentionally indicate to the subject the kind of response indicate to the subject the kind of response desired. desired.

5555

Data collection methodologyData collection methodology

Data obtained through structured Data obtained through structured interviews may differ qualitatively from interviews may differ qualitatively from data gathered through a test. It would be data gathered through a test. It would be difficult to compare the findings of these difficult to compare the findings of these studies since each approached the studies since each approached the problem in a different way. problem in a different way.