ICAO Circular 318

© EUROCONTROL 2010

ICAO – Regional Workshop on Language Proficiency Requirements Implementation

Rome, March 3rd to 5th 2010

ICAO Circular 318ICAO Circular 318

Adrian EnrightAdrian Enright

EUROCONTROLEUROCONTROL


Rome, March 3rd to 5th 2010 © EUROCONTROL 2010

ICAO Circular 318ICAO Circular 318

Gives more detailed guidanceGives more detailed guidance Has recommended criteria for selection of Has recommended criteria for selection of

aviation language testing programmesaviation language testing programmes An indispensable guideline to CAA regulators, An indispensable guideline to CAA regulators,

aircraft operators and air navigation service aircraft operators and air navigation service providersproviders



Captain? Captain can you hear me? …have you found my copy of ICAO

Circular 318?

An indispensable guideline to aircraft operators …An indispensable guideline to aircraft operators …



Circular 318Circular 318

ICAO LPR apply to ICAO LPR apply to allall languages, but languages, but Circular 318 focuses on English as the language Circular 318 focuses on English as the language

for which most training and testing programmes for which most training and testing programmes need to be developedneed to be developed

Circular 318 will not replace the extensive Circular 318 will not replace the extensive standards, guidelines and principles found in the standards, guidelines and principles found in the literature on language testingliterature on language testing



BackgroundBackground

1998 321998 32ndnd ICAO Assembly ICAO Assembly 2003 comprehensive set of SARPs2003 comprehensive set of SARPs

● Annex 1 Personnel LicensingAnnex 1 Personnel Licensing● Annex 6 Operation of AircraftAnnex 6 Operation of Aircraft● Annex 10 Aeronautical CommunicationsAnnex 10 Aeronautical Communications● Annex 11 Air Traffic ServicesAnnex 11 Air Traffic Services● PANS-ATMPANS-ATM

March 5March 5thth 2011 final deadline for implementation of LPRs 2011 final deadline for implementation of LPRs



Fundamental test principlesFundamental test principles

ValidityValidity ReliabilityReliability PracticabilityPracticability



Test purposesTest purposes

DiagnosticDiagnostic PlacementPlacement ProgressProgress AchievementAchievement AptitudeAptitude ProficiencyProficiency



Test focusTest focus

Speaking and listening proficiency Speaking and listening proficiency High stakes testHigh stakes test Work related contextWork related context ICAO Rating Scale, andICAO Rating Scale, and Holistic descriptorsHolistic descriptors



Use of ICAO phraseologyUse of ICAO phraseology

Testing phraseology is a operational Testing phraseology is a operational issue, butissue, but

Use of phraseology in a work-related Use of phraseology in a work-related context is acceptablecontext is acceptable

Can be used as a warm-up, orCan be used as a warm-up, or In setting the contextIn setting the context



Recommended criteriaRecommended criteria

Self-contained statements with supplementary Self-contained statements with supplementary informationinformation● What it meansWhat it means● Why it is importantWhy it is important● Additional informationAdditional information



Test Design and ConstructTest Design and Construct Tests should address the specific and unique Tests should address the specific and unique

requirements of aviation language testingrequirements of aviation language testing A separate listening test can provide independent A separate listening test can provide independent

information about comprehensioninformation about comprehension Tests should be specific to aviation operations – Tests should be specific to aviation operations –

● Visual AND non-visual communicationsVisual AND non-visual communications● customize tests for pilots or controllerscustomize tests for pilots or controllers

Language proficiency tests should not be designed to Language proficiency tests should not be designed to evaluate technical knowledgeevaluate technical knowledge

The final score is the LOWEST of the six rated criteriaThe final score is the LOWEST of the six rated criteria



RatingRating

Rating should be carried out by a minimum of Rating should be carried out by a minimum of two raterstwo raters

An independent 3rd rater when the 2 raters An independent 3rd rater when the 2 raters cannot agree between level 4 and level 3cannot agree between level 4 and level 3



Test familiarisationTest familiarisation

Complete sample test freely available Complete sample test freely available Helps to ensure that test takers are not surprised Helps to ensure that test takers are not surprised

by the format of the test and are properly able to by the format of the test and are properly able to demonstrate their language proficiencydemonstrate their language proficiency

Test description or a marketing brochure is not Test description or a marketing brochure is not sufficientsufficient



Test securityTest security

Test security is essential for ensuring reliability Test security is essential for ensuring reliability of, and confidence in, the testing systemof, and confidence in, the testing system

A full description of security measures should be A full description of security measures should be available to all decision makersavailable to all decision makers



Testing TeamTesting Team

Recommended 2 ratersRecommended 2 raters● language specialist expertise language specialist expertise ● operational expertiseoperational expertise● Should demonstrate at least the highest level of Should demonstrate at least the highest level of

proficiency expected of test takersproficiency expected of test takers

Familiar with aviation English vocabulary and Familiar with aviation English vocabulary and structuresstructures

Complete recurrent training at least once a yearComplete recurrent training at least once a year



Rater reliabilityRater reliability

Intra-rater reliabilityIntra-rater reliability● Extent to which a particular rater consistently uses a Extent to which a particular rater consistently uses a

proficiency scaleproficiency scale

Inter-rater reliabilityInter-rater reliability● Extent to which a group of independent raters agree Extent to which a group of independent raters agree

on a test taker’s performanceon a test taker’s performance



ChecklistChecklist

Chapter 3 provides a simple checklistChapter 3 provides a simple checklist Cross-referenced to the sub-paragraphsCross-referenced to the sub-paragraphs Assist States in determining:Assist States in determining:● how far they have gonehow far they have gone● how far they have still to gohow far they have still to go

in implementing valid and reliable language in implementing valid and reliable language proficiency testsproficiency tests



What? That’s not the pre-flight checklist…

Is the rating process documented?



GlossaryGlossary

Useful glossary of testing terms and acronymsUseful glossary of testing terms and acronyms ““Everything you wanted to know about testing Everything you wanted to know about testing

but were afraid to ask”but were afraid to ask” Not quite, but a good start.Not quite, but a good start.



Let us not forget just one reason why we set off on this journey to improve aviation English

Tenerife March 1977



A final thought …A final thought …

We We havehave come a long way in the last 12 years in come a long way in the last 12 years in improving language proficiency in aeronautical improving language proficiency in aeronautical communication, but the journey is not over yet. communication, but the journey is not over yet. Our commitment today is in laying the foundation Our commitment today is in laying the foundation for generations to come.for generations to come.

Documents

ICAO Circular 318