1

TESL 4340 -- Evaluating CALL Packages:Curriculum/Pedagogic

al/Linguistics

Dr. Henry Tao GUO 郭涛Email: henryguo@uic.edu.hkOffice: B 418

2

Outline

• Part one: Language program evaluation (Lynch 1996: 1-9)

• Part two: Evaluating CALL(Chapelle 2001:44-94)

3

Language program evaluation

• Definition• Critical issues• Context-adaptive model (CAM)

4

Definition: evaluation

• Evaluation here is defined as the systematic attempt to gather information in order to make judgments or decisions.

• As such, evaluative information can be both qualitative and quantitative in form and can be gathered through different methods such as observation or the administration of pencil-and-paper tests.

5

CAM

• Audience and goals• Context inventory• Preliminary thematic framework• Data collection design/system• Data collection and analysis• Evaluation report

(Lynch 1990a)

6

7

Audience and goals

Audience• Who is requesting

the evaluation? • Who will be

affected by the evaluation?

Stakeholders (or clients)

& Peripheral audience

Goals• Why is the

evaluation being conducted?

• What information is being requested and why?

The goals of evaluation may be

determined by audience

8

• The role of evaluator is determined by the particular audience and their goals

Someone providing consultation An expert standing in judgment A collaborator in program development A decision-making facilitator

• Hiring external evaluators for greater objectivity or internal evaluators for improving the curriculum?

9

Context inventory

1. Availability of a comparison group2. Availability of reliable and valid measures

of language skills3. Availability of various types of evaluation

expertise4.Timing of the evaluation5.The selection process for admitting

students into the program

10

Critical issues

• Why CALL programs should concern themselves with program evaluation?

• Internal quest for program improvement

• Externally imposed requirement in order to justify program funding

11

6. Characteristics of the program students7. Characteristics of the program staff8. Size and intensity of the program9. Instructional materials and resources10. Perspective and purpose of the program11. Social and political climate

12

Preliminary thematic framework

• Where should the evaluator being?

• What aspects of the program should evaluator investigate in detail?

13

A preliminary thematic framework provides a conceptualization of the program in terms of the salient issues and themes that have emerged from the determination of audience and goals and elaboration of the context inventory.

Articulating this framework provides the evaluator with a focus that will guide the collection and analysis of evaluation data.

14

An example of PTF developed for EST reading program

1. Effects of focusing instruction on reading only

2. Effects of focusing instruction on reading skills and strategies

3. Effects of using authentic reading texts4. Feasibility of using Spanish versus English

for instruction

15

5. Availability of classrooms6. Feasibility of using a ‘modified adjunct

model’ approach7. Feasibility and effects of conducting

classroom-centered research8. Levels of student proficiency in English

upon entering the program (Adapted from Lynch 1990a)

16

Data collection design/system

• What type of data need to be gathered-quantitative, qualitative or both?

• What will be the best method for gathering the data?

• The context inventory can be used to determine the feasibility of certain types of data collection design.

17

18

Data collection and analysis

• Appropriate conduct of the data-gathering procedures and the interpretation of the results.

In the case of quantitative designs, have the assumptions of the design and statistical models been met?

In the case of qualitative design, have the procedures for data gathering been portrayed accurately, and have alternative interpretations of the data been pursued?

19

Evaluation report

• The evaluator must be sensitive to the audience and goals of the evaluation

• The social and political climate dimension of the context inventory needs to be considered carefully at this stage

20

• The critical issue is how to communicate the findings of the evaluation honestly and successfully.

• Prepare multiple reports or express the findings in different ways, depending on the intended audiences.

21

Evaluating CALL

• The problem of instructed SLA• Principles for CALL evaluation: three

needs• Judgmental evaluation of CALL• Empirical evaluation of CALL

22

The problem of instructed SLA

• Cognitive conditions for SLA• Social-affective conditions for SLA• Other factors

23

Cognitive conditions for SLA

Choose a range of target structures Choose tasks which meet the utility

condition Select and sequence tasks to achieve

balanced goal development Maximize the chances f focus on form

through attentional manipulation Use cycles of accountability

24

Social-affective conditions for SLA

Willingness to communicate (WTC)

The desire to communicate with a particular person

Communicative self confidence at that particular moment

Interpersonal motivation Intergroup motivation

25

Self confidence Intergroup attitudes(e.g., integrativeness)Social situation(i.e.,features of context

affecting communication)Communicative competence Intergroup climatepersonality

26

Other factors

Individual differences in cognitive characteristics of learners due to, for example, age or cognitive style.

Learning situation such as the effects of task choice on teachers and learners and others who may be involved with the task

Practical factors such as the available resources

27

Principles for CALL evaluation: three needs

• First, evaluation criteria should incorporate findings and theory-based speculation about ideal conditions for SLA such as those outlined above.

• Second, criteria should be accompanied by guidance as to how they should be used; in other words, a theory of evaluation needs to be articulated.

• Third, both criteria and theory need to apply not only to software, but also to the task that the teacher plans and that the learner carries out.

28

Evaluation as an argument

29

Judgmental and empirical analyses

30

Criteria from theory and research on SLA

31

Criteria applied based on task purpose

• These criteria for CALL appropriateness need to be applied in view of the purpose of a CALL task.

• Skehan's discussion of performance goals for task, including learners‘ fluency,accuracy, and complexity, might be augmented by comprehension goals and goals concerning pragmatic competence (Kapser,1997).

32

• Moreover, tasks may have different purposes at various stages of instruction (Doughty & Williams,1998).

• Whatever the goal of the CALL task, however, evaluation of the task requires that it have a stated purpose.

33

The centrality of language learning

• Even though the importance of each of the six criteria may vary depending on the purpose of the tasks, language learning potential should be considered the most critical for CALL activities

34

Judgemental evaluation of CALL

35

36

Cases(Chapelle 58-67)

• Computer-assisted classroom discussion• Microworld• Text analysis• Storyboard• Concordancing

37

Summary

38

Empirical evaluation of CALL

• Evidence for CALL qualities

39

• ‘Students are often doing soimthing very different from what [ language teachers ] assume they are doing’ ( Hosenfeld,1976:123). In other words, it is necessary to identify the observable data that provide evidence of CALL qualities.

40

Evidence for CALL qualities

41

42

Language learning potential

• Focus on form

• Modified interaction

• Modified output

• Research methods

43

Learner fit

• Level of linguistic difficulty

• Individual difference

• Research methods

44

Meaning focus

• Effects of meaning-based instruction

• Assessing engagement with meaning

45

Authenticity

• Comparing CALL with non-CALL activities

• Research methods

46

Positive impact

47

48

Practicality

• Given the role of resources for the success of CALL, some formal mechanism needs to be in place to monitor adequacy, and an argument about CALL appropriateness should include a statement about sufficiency of resources.

49

Conclusion

• Evidence concerning the six ideal qualities of a particular CALL task needs to be combined to form an evaluative argument about the appropriateness of a CALL task for particular learners at a given point in time.

50

• Integrated research is needed examine the types of CALL activites to seek evidence about CALL appropiratness in particular settings.

51

• From such research, implications can be drawn for specific contexts, and eventually research on particular types of activities might be generalised in a way that can inform general principles of CALL.