Into Europe - European Standards in Language Assessment Conference

9-10 February 2006

The BGF Linking ExperienceBánóczy ErikaBenke Eszter

Budapesti Gazdasági FőiskolaNyelvvizsga és Továbbképző Központ

Harmonizing national examinations with the Common European Framework of Reference (CEF)

Background to the project• The origins of the existing level system• The Strasbourg project

Case Studies (WIP)• Objectively scored tasks• Subjectively scored tasks

Examiners’ familiarity with the CEF LSP and the CEF

BackgroundLanguage examination levels according to the Accreditation manual (1999), p.36.

Common European Framework Levels

Levels in the examination system

Language examination levels in

Hungary

C2 advanced

C1

B2 intermediate

B1 elementary

A2

A1

levels not eligible for accreditation

Background

Language examination levels according to the Accreditation manual (1999), p.33.

Intermediate language proficiency corresponds to the intermediate level as suggested by the Council of Europe intermediate level (Vantage Level) and extends into the lower part of advanced level (Operational Proficiency).

Language examination levels accredited in 2000

Common European Framework Levels

Language examination levels in

Hungary

C2 advanced

C1

B2 intermediate

B1 elementary

A2

A1

levels not eligible for accreditation

BackgroundModified language examination levels according to the new Accreditation manual (2004), p.25.

Accredited language examinations

Levels in the examination system

Council of Europe levels

Mastery (C2)

Advanced

Effective Operational Proficiency (C1)

Intermediate

Vantage (B2)

Elementary

Threshold (B1)

Not eligible for accreditation Waystage (A2)

Not eligible for accreditation

Breakthrough (A1)

BackgroundLanguage examination levels according to the Accreditation manual (2004) p.8.

„The state-accredited examination system intends to harmonize the theory and practice of national and international (European) language examinations. This effort is manifest … in the transferability of the Hungarian three-level system and the more detailed and up-to-date level system of the Council of Europe. (Work towards harmonization is still in progress.)

Levels to be harmonized

C2 Mastery C1 EOP B2 Vantage B1 Threshold A2 Waystage A1 Breakthrough

Advanced

Intermediate

Elementary

?

?

?

Aim of the BGF project

Piloting the Manual Empirical validation of original levels Training of experts Modification of existing levels Benchmarking existing performance samples Creating new tasks based on new levels

Seminal documents and materials

Relating Language Examinations to the Common European Framework of Reference for Languages: learning, teaching, assessment. Manual (2003)

Reference Supplement (2004) Nyelvvizsgák illeszkedése a Közös Európai

Referenciakerethez (2005) Case Studies Calibrated performance samples

Procedures to relate examinations to the CEF

Familiarisation with the CEF Specification Standardisation Empirical validation Reporting the results

Work completed so far

3 stages• English project

• participants (anchor persons)• procedures

• German project• large-scale examiner training

time and materials used

Breakdown of working hours in the first stage (1)

Type of work total Contact hours 6 sessions 23 hrs Individual work ( x14 persons) watching calibrated samples1 2x180 mins 6 hrs evaluation of own oral performance samples according to the CEF levels and according to own assessment scales

14x (20+10) 30 mins

7 hrs evaluation of own written performance samples according to the CEF levels and according to own assessment scales

2x14x10 mins

4,6 hrs Evaluation of BEC written taks 6x10 mins 1 hrs Marking and data analysis of reading tasks2 2x1,5 hrs 3 hrs Total 44,6 hrs

1 It seemed necessary to watch the calibrated samples again right before team members started individually rating the own performance samples of the examination board. 2 The reading project will be fully described in the case study

Breakdown of working hours in the first stage (2)Data analysis data entry and analysis for written tasks 30 hrs data entry and analysis for written tasks 70 hrs Summary, discussion of results 20 hrs Total 120 hrs Workshop preparation 1st workshop 20 hrs 2nd workshop 2 hrs 3rd workshop 2 hrs 4th workshop 15 hrs 5th workshop 3 hrs 6th workshop 1 hrs Total 43 hrs Technical support videorecording of pilot oral exams 6 hrs digitalization of videorecordings 6 hrs copying videorecordings 90 hrs Total 102 hrs

Technical equipment used

Type of work Technical equipment Presentations laptop+projector Standardization – evaluation of sample oral performances

laptop+projector+loudspeakers

Validation of tasks computer lab Registering assessments, discussions1 digital dicataphones Recording pilot oral examinations digital camera + external

microphone Preparing handouts and documentation

Photocopier, binding machine

1 With participants’ permission

Objectively scored tasks

Familiarisation with CEF and DIALANG scales Study of calibrated sample tasks (if and where

available) Standard setting – modified Angoff method Collation and tabulation of results Comparison of qualitative and quantitative data

Standard-setting grid for a reading comprehension task

CEF Overall reading comprehension

(p.69.)

item1

item2

item3

item4

item5

Data collection 1itemszám Balázs Gergely Réka Rita Ancsa Erika Mari Ági Ági2 Anikó Anikó2 Fániitem1 B1 B2 B2 B2 B1 B1 B1 B2 B2 B2 B2 B1item2 B1 B1 B2 B1 B1 B1 B1 B2 B2 B2 B2 A2item3 B1 B1 B2 B2 B1 B1 B1 B1 B2 C1 C1 A2item4 B2 B1 B2 B1 A2 B1 B2 B1 B2 C1 C1 A2item5 B1 B2 B2 B2 B1 B1 B1 B1 B2 B1 B2 A2item6 B1 B2 B1 B2 A2 B2 B2 B1 B2 B1 B1 A2item7 B2 B2 B2 B2 B1 B2 B1 B2 B2 B1 B1 A2item8 C1 B1 B2 B2 B1 B2 B2 B2 B2 B1 B1 B2item9 B2 B1 B2 B2 B1 B2 B2 B2 B2 B1 B1 A2item10 B2 B2 B2 B2 B2 B2 B2 B2 B2 B2 B2 B2item11 C1 B2 B2 B2 A2 B2 B2 B2 B2 B1 B1 B2item12 B1 B2 B1 B2 A2 B1 B2 B1 B2 B2 B2 A2item13 C1 B2 C1 B2 A2 B1 B2 B1 B2 B2 B2 A2item14 B2 B2 B1 B2 A2 B1 B2 B1 B2 B2 B2 A2item15 B2 B2 B1 B2 A2 B1 B2 B1 B2 B2 B2 A2item16 B1 B2 B1 B2 A2 B1 B2 B2 B2 B2 B2 A2item17 B2 B2 B2 B2 B1 B2 B2 B2 B2 B2 C1 B2item18 B2 B2 B2 B2 B2 B2 B2 B2 B2 B2 B2 B2item19 B2 B2 B2 B2 B2 B2 B2 B2 B2 B1 B1 B1item20 B2 B2 B2 B2 B1 B2 B2 B1 B2 B2 B2 B2

Data collection 2item No A1 A2 B1 B2 C1 C2 Final

item1 5 6 B2 item2 1 5 5 B1 item3 1 6 2 2 B1 item4 2 3 4 2 B2 item5 1 6 4 B1 item6 2 5 4 B1 item7 1 4 6 B2 item8 4 6 1 B2 item9 1 4 6 B2 item10 11 B2 item11 1 2 7 1 B2 item12 2 4 5 B1 item13 2 2 5 2 B2 item14 2 3 6 B2 item15 2 3 6 B2 item16 2 3 6 B2 item17 1 9 1 B2 item18 11 B2 item19 3 8 B2 item20 2 9 B2 Total: 20 60 120 9 B2

Data analysis

Method applied • Comparison of qualitative and quantitative data

item No A1 A2 B1 B2 C1 C2 Final facility value

% logit

item1 5 6 B2 83 -0,82 item2 1 5 5 B1 59 0,63 item3 1 6 2 2 B1 93 -1,91 item4 2 3 4 2 B2 80 -0,6 item5 1 6 4 B1 94 -2,08 item6 2 5 4 B1 85 -1 item7 1 4 6 B2 86 -1,09 item8 4 6 1 B2 75 -0,33 item9 1 4 6 B2 83 -0,75 item10 11 B2 66 0.25 item11 1 2 7 1 B2 85 -1,09 item12 2 4 5 B1 70 0.09 item13 2 2 5 2 B2 23 2.61 item14 2 3 6 B2 64 0.42 item15 2 3 6 B2 55 0.78 item16 2 3 6 B2 58 0.68 item17 1 9 1 B2 58 0.73 item18 11 B2 71 -0,03 item19 3 8 B2 40 1,56 item20 2 9 B2 35 1.95 Total: 20 60 120 9 B2

Problematic item

Item No CEF levelFacility value

logit

item 11 B2 85% -1.09 .787

Alpha

if item deleted

Alpha = Alpha = ..771771

External validation

Criterion test• BEC Vantage level• Content analysis to establish comparability – (ALTE

content analysis checklist) Correlation

• comparison of statistical data

Results – descriptive statisticsBEC

Vantage(N=107)

NYTKIntermediate

(N= 106)

Mean

SDS.E. mean

11.18

2.98.288

10.84

3.422.332

Mean facility value61.99% 54.24%

Cronbach alpha 0.66 0.65

Results - correlation

r=0.59 (p< 0.01) after correction for attenuation r=0.9 considerable similarity between empirical and

intuitively set levels of difficulty

Subjectively scored tasks

Oral proficiency tests• watching videotaped calibrated performance samples• benchmarking own samples• comparison of CEF based and original scores

Writing• benchmarking own samples• comparison of CEF based and original scores

Writing – selection of criteria

Original local assessment criteria

CEF criteria judged as closely corresponding to the original

task achievement overall written production (p.61) vocabulary vocabulary range, vocabulary control (p.112) coherence and cohesion coherence and cohesion (p.125.) grammar, accuracy grammatical accuracy ( p.114)

Results so far

benchmarked intermediate performance samples in six languages

accumulated experience and emerging expertise – fairly thorough familiarity with the CEF level system

Survey on familiarity with the CEF 82 respondents

• 35 English• 21 German• 10 French• 10 Spanish• 4 Italian• 2 Russian

18 yrs average teaching experience questionnaire

• 15 statements• 5 point Likert-scale

Familiarity with the CEF - questionnaire

I have heard of the CEF before. I know the Hungarian version of the CEF. I know the language version of the CEF I teach. I have my own copy of the CEF. I know the level system used in the CEF. I am familiar with the descriptors used in the

CEF.

Use of the CEF - questionnaire

I use the CEF for defining proficiency levels. I use the CEF in curriculum design. I use the CEF letter level system when choosing course

books. I use the diagnostic self-assessment scales.

The benchmarking workshop and the future - questionnaire

I find the sorting tasks useful in getting familiar with the CEF levels.

The CEF descriptors are easy to apply. It is possible to realistically assess students’ language proficiency

with the help of the CEF scales. I find it feasible to harmonize our existing system of levels with the

CEF levels. The CEF levels are becoming more and more common in the

professional discourse/communication of those involved in education (teachers, students, parents, employers etc.).

Top 3

• I find the sorting tasks useful in getting familiar with the CEF levels. ( =4.1)

• I know the level system used in the CEF. ( =3.9)• I find it feasible to harmonize our existing system of

levels with the CEF levels. ( =3.8) X

XX

Bottom 3

• I use the CEF in curriculum design. ( =2.3)

• I have my own copy of the CEF. ( =2.3)

• I use the diagnostic self-assessment scales. ( =2.2)

X

X

X

ESP and the CEF

ESP and the CEF

Some intriguing issues: L(SP) competence definitions of LSP

• Swales, 1985• Strevens, 1988• EAP-EOP (Robinson, 1991)• general-specific continuum (Dudley-Evans & St John,

1998)

ESP and the CEF

Some intriguing questions:

Which ESP/LSP definition is our system based on?How are ESP/LSP skills different from EGP skills?

ESP and the CEF

General purpose language testing vs LSP testing:

• authenticity of task

• interaction between language knowledge and specific purpose content knowledge

(Douglas, 2000)

ESP and the CEF

specific aspect of the specifications (‘specific specifications’)

specific aspect of the operationalisation of the construct

specific aspect of the tasks

ESP and the CEF

special*job*

field*vocation*

terminology*

ESP and the CEF Table 1. Common Reference Levels: global scale ( p.24)

B2 Can understand the main ideas of complex text on both concrete and abstract topics, including technical discussions in his/her field of specialisation.

ESP and the CEF Table 2. Common Reference Levels: self-assessment grid (pp. 26-27)

A2 Spoken Production I can use a series of phrases and sentences to describe in simple terms … my

present or most recent job.B1 Spoken Interaction

I can enter unprepared into conversation on topics that are familiar … (e.g. work).C1 Reading

I can understand specialized articles … even when they do not relate to my field.C2 Reading

I can read with ease virtually all form of written language …such as … specialised articles.

ESP and the CEF

Sustained monologue (p.59)A2 Can describe his/her family, living conditions,

educational background, present or most recent job.Creative writing (p.62)B1 Can write about everyday aspects of his/her

environment, e.g. … a job… in linked sentences.

ESP and the CEF

Overall listening comprehension (p.66)

B2 Can understand standard spoken language, live or broadcast, on both familiar and unfamiliar topics normally encountered in personal, social, academic or vocational life.C1 Can understand enough to follow extended speech on abstract and complex topics beyond his/her own

field …

ESP and the CEFListening as a member of a live audience (p. 67)C2 Can follow specialised lectures and presentations employing a high

degree of colloquialism, regional usage or unfamiliar terminology.Overall reading comprehension (p.69)A2 Can understand short, simple texts on familiar matters of a concrete

type which consist of high frequency everyday or job-related language.B1 Can read straightforward factual texts on subjects related to his/her

field and interest with a satisfactory level of comprehension.C1 Can understand in detail lengthy, complex texts, whether or not they

relate to his/her own area of speciality, provided he/she can reread difficult sections.

ESP and the CEFReading for information and argument (p.70)B2 Can obtain information, ideas and opinions from highly specialised

sources within his/her field. Can understand specialised articles outside his/her field, provided he/she can use a dictionary occasionally to confirm his/her interpretation of terminology.

Reading instructions (p.71)B2 Can understand lengthy, complex instructions in his field, including

details on conditions and warnings, provided he/she can reread difficult sections.C1 Can understand in detail lengthy, complex instructions … whether or

not the instructions relate to his/her own area of speciality…

ESP and the CEF

Overall listening comprehension (p.66)B1 Can understand straightforward factual information

about common everyday or job related topics…B2 Can understand standard spoken language, live or

broadcast, on both familiar and unfamiliar topics normally encountered in personal, social, academic or

vocational life.C1 Can understand enough to follow extended speech on

abstract and complex topics beyond his/her own field

ESP and the CEF

Understanding a native speaker interlocutor (p.75)C1 Can understand in detail speech on abstract and

complex topics of a specialist nature beyond his/her own field …

C2 Can understand any native speaker interlocutor, even on abstract and complex topics of a specialist nature beyond his/her own field …

ESP and the CEF

A2 B1 job, work,

B2 C1 own field, speciality, specialised,

C1 C2 beyond own field

Useful/beneficial experience

Internal validity of tests is vital for external validation

• continuous internal validation

Ongoing local and global validation

Useful/beneficial experienceApparent need for

• intensive familiarization• adequate number of experts• a keen eye for the differences between the original

and the translated versions of the CEF• harmonization within and across languages

Further challenges• bilingual tasks• LSP examinations

C2 Mastery

C1 Effective Operational Proficiency

B2 Vantage

B1 Threshold

A2 Waystage

A1 Breakthrough

Cooperation and sharing of Cooperation and sharing of experience to make it common.experience to make it common.

May we all continue to learn from each other.

Thank you.

banoczy@bgf.nyelvvizsgak.hubenke@bgf.nyelvvizsgak.hu