ASSESSING GRADE 5 PUPIL PERFORMANCE (Reading and Mathematics in 2001) Socialist Republic of Vietnam

ASSESSING GRADE 5 PUPIL ASSESSING GRADE 5 PUPIL PERFORMANCEPERFORMANCE

(Reading and Mathematics(Reading and Mathematicsin 2001)in 2001)

Socialist Republic of VietnamSocialist Republic of Vietnam

VietnamVietnam

Assessing Grade 5 Reading & MathematicsAssessing Grade 5 Reading & Mathematics

Presentation OutlinePresentation Outline

Why do the study -- goals?Why do the study -- goals? The research designThe research design Different stages in the studyDifferent stages in the study Teams and organizationTeams and organization Conduct of studyConduct of study FindingsFindings

Why do the study -- goals?Why do the study -- goals? The research designThe research design Different stages in the studyDifferent stages in the study Teams and organizationTeams and organization Conduct of studyConduct of study FindingsFindings

VietnamVietnam


Why do the study?Why do the study? Fall in demographic Fall in demographic

demand allows the demand allows the Govt.’s concern to Govt.’s concern to move from move from AccessAccess to a focus on to a focus on QualityQuality

But what is current But what is current status of Quality and status of Quality and how can it be how can it be measured?measured?

Population 6-11

56789

1011

2001 2004 2007 2010

Years

Mill

ions Population 6-11

VietnamVietnam


Study goalsStudy goals

A clear measurement of educational quality and A clear measurement of educational quality and learning outcomes...learning outcomes...

to document the baseline at the turn of the to document the baseline at the turn of the centurycentury

to allow measurement of the incremental impact to allow measurement of the incremental impact of all future interventions in primary education of all future interventions in primary education (new curriculum; in-service teacher training; (new curriculum; in-service teacher training; additional reforms in instructional methods)additional reforms in instructional methods)

VietnamVietnam


Study QuestionsStudy Questions

What is the level of achievement of Grade 5 pupils What is the level of achievement of Grade 5 pupils overall and in the areas of reading and mathematics? overall and in the areas of reading and mathematics?

What is the level of Grade 5 teachers in these subjects?What is the level of Grade 5 teachers in these subjects? What are the characteristics of pupils; teachers; What are the characteristics of pupils; teachers;

classrooms; materials; schools?classrooms; materials; schools? Are predefined BENCHMARKS being met?Are predefined BENCHMARKS being met? Have inputs been allocated to schools equitably?Have inputs been allocated to schools equitably? What variables are associated with difference between What variables are associated with difference between

the most and least effective schools?the most and least effective schools? What variables are most associated with achievement?What variables are most associated with achievement?

VietnamVietnam


The Study DesignThe Study Design

A cross-sectional sample surveyA cross-sectional sample survey Aimed at Grade Five pupils (final grade of Aimed at Grade Five pupils (final grade of

Primary)Primary) Testing learning outcomes in two subject Testing learning outcomes in two subject

areas: Reading (in Vietnamese) and areas: Reading (in Vietnamese) and MathematicsMathematics

With a sample that is representative at With a sample that is representative at both the NATIONAL and the PROVINCIAL both the NATIONAL and the PROVINCIAL levels (61 provinces)levels (61 provinces)

VietnamVietnam


The Eight Stages of the StudyThe Eight Stages of the Study

1.1. Design of the StudyDesign of the Study2.2. Trial and AdjustmentTrial and Adjustment3.3. Preparation for & Collection of DataPreparation for & Collection of Data4.4. Data PreparationData Preparation5.5. Data AnalysisData Analysis6.6. Report WritingReport Writing7.7. Policy Action and DisseminationPolicy Action and Dissemination8.8. Follow-upFollow-up

VietnamVietnam


1.1. Study DesignStudy Design

Define the Define the policy questionspolicy questions Prepare the Prepare the analysis plananalysis plan with empty tables with empty tables Design the Design the questionnairesquestionnaires Develop Develop teststests in reading and maths for pupils and in reading and maths for pupils and

teachersteachers Define the Define the samplesample Construct the Construct the instrumentsinstruments Design the process to administer the instruments Design the process to administer the instruments

& & write data collection manualwrite data collection manual

VietnamVietnam


2.2. Trial & AdjustmentTrial & Adjustment

Print InstrumentsPrint Instruments Train data collectorsTrain data collectors Administer data collection in trial areaAdminister data collection in trial area Prepare data & check the test results Prepare data & check the test results

of the trial itemsof the trial items Revise instruments and proceduresRevise instruments and procedures

VietnamVietnam


3. Preparation for & Collection of Data3. Preparation for & Collection of Data

Set up the data supervisory systemSet up the data supervisory system Hold provincial-level data supervisor Hold provincial-level data supervisor

training workshopstraining workshops Organize local-level data collector trainingOrganize local-level data collector training Explain sampling techniquesExplain sampling techniques Supervise both sampling and data Supervise both sampling and data

collectioncollection Return all instruments to central officeReturn all instruments to central office

VietnamVietnam


4. Data preparation4. Data preparation Define the editing rulesDefine the editing rules Carry out hand editing of data bookletsCarry out hand editing of data booklets Data entry and verification (double Data entry and verification (double

entry for 2 schools/province ~3%)entry for 2 schools/province ~3%) Data cleaning and consistency checks Data cleaning and consistency checks

(video training from IIEP)(video training from IIEP) Sample weighting of the data setSample weighting of the data set Prepare derived variablesPrepare derived variables

VietnamVietnam


5.5. Data AnalysisData Analysis

Test scoring and Item analysisTest scoring and Item analysis Statistical analysis and preparation of Statistical analysis and preparation of

the tables (c.f. #2)the tables (c.f. #2) Cross-tabulations and checksCross-tabulations and checks Correlation and regression analysisCorrelation and regression analysis Effective school analysisEffective school analysis

VietnamVietnam


6.6. Report WritingReport Writing

Interpretation of the tablesInterpretation of the tables Preparation of the text and descriptive Preparation of the text and descriptive

and analytical sectionsand analytical sections Review results against other existing Review results against other existing

informationinformation Discussion of results by Govt./MoET Discussion of results by Govt./MoET

managerial groupmanagerial group

VietnamVietnam


7.7. Policy Actions & DisseminationPolicy Actions & Dissemination

Preparation of policy actions, recommendations Preparation of policy actions, recommendations and changesand changes

Review and adoption by MoET and Govt. Review and adoption by MoET and Govt. leadershipleadership

Publication of resultsPublication of results Preparation of regional (3) workshopsPreparation of regional (3) workshops Use of results in the annual meeting of Provincial Use of results in the annual meeting of Provincial

Education DirectorsEducation Directors Series of provincial level dissemination workshopsSeries of provincial level dissemination workshops

VietnamVietnam


8.8. Follow-upFollow-up

Prepare a full archival datasetPrepare a full archival dataset List all additional research topicsList all additional research topics Start supplementary analysis of the dataStart supplementary analysis of the data Write up recommendations for inclusion Write up recommendations for inclusion

in the next national assessment exercisein the next national assessment exercise– Sampling; Focus; Level; Subjects;Sampling; Focus; Level; Subjects;– Corrections; AdditionsCorrections; Additions

VietnamVietnam


The TeamsThe Teams Overall ManagementOverall Management Item designItem design Questionnaire designQuestionnaire design SamplingSampling Supervision of Data collectionSupervision of Data collection Data collectorsData collectors Data EditingData Editing Data EntryData Entry Data Cleaning & VerificationData Cleaning & Verification Data AnalysisData Analysis

VietnamVietnam


The Teams The Teams (continued)(continued)

Results & Writing TeamResults & Writing Team Ministry Review teamMinistry Review team Dissemination TeamDissemination Team Supplemental analysis teamsSupplemental analysis teams Policy recommendation teamPolicy recommendation team

VietnamVietnam


Organizational StructureOrganizational Structure

REGULARREVIEW OF

RESEARCH FINANCINGMANAGEMENT RESULT DESIGN

SAMPLE DESIGN

DISSEMINATION

MATHS READING SCHOOL TEACHER PUPIL EDITING ENTRY CLEANING STATISTICAL REPORT WRITING

Local Level Data collection

TRAINERS & SUPERVISORS

DATA PREPARATION ANALYSIS TEAM

PROVINCIAL DATA COLLECTION TEAMS (61)

TECHNICAL TEAMS (7)

ITEM DESIGN QUESTIONAIRE DESIGN DATA COLLECTION PROCEDURES

OVERSIGHT & CONTROL

EXTERNAL SUPPORTMANAGEMENT &ADMINISTRATION

VietnamVietnam


Conduct of the StudyConduct of the Study Design early 2000Design early 2000 Sample August 2000Sample August 2000 Collect data April 2001Collect data April 2001 Data entry July/Oct 2001Data entry July/Oct 2001 Cleaning Oct/Dec 2001Cleaning Oct/Dec 2001 Analysis Jan/May 2002Analysis Jan/May 2002 Report writing April/June 2002Report writing April/June 2002 Dissemination July 2002+Dissemination July 2002+

VietnamVietnam


ResultsResults

VietnamVietnam


Planned and achieved samplePlanned and achieved sample

Planned Achieved Rate (%)School 3639 3635 99.89Teachers 7278 7178 98.63Pupils 73200 72660 99.26

VietnamVietnam


Hours/week teaching in primary schoolsHours/week teaching in primary schools

France 26

Ireland 25

New Zealand 25

Portugal 24

Spain 23.8

Netherlands 23.3

Sw itzerland 23

Korea, Republic of 22.6

Australia 22.2

Italy 21.2

Germany 20.1

Greece 19.5

Norw ay 18.8

Japan 18.1

Vietnam: 16.7 hours per weekVietnam: 16.7 hours per week

VietnamVietnam


Total school resources by school locationTotal school resources by school location

0.0

2.0

4.0

6.0

8.0

10.0

12.0

14.0

16.0

18.0

Isolated Rural Urban

School location

Res

ourc

es

VietnamVietnam


Total school resources by regionTotal school resources by region

0.0

2.0

4.0

6.0

8.0

10.0

12.0

14.0

16.0

Red RiverDelta

Northeast Northw est NorthCentral

CentralCoast

CentralHighlands

Southeast MekongDelta

Regions

Res

ourc

es

VietnamVietnam


Classroom resources by regionClassroom resources by region

-1

-0.8

-0.6

-0.4

-0.2

0

0.2

0.4

0.6

0.8

Regions

Sta

nd

ard

ise

d S

co

res

Classroom furniture

Teaching Materials

VietnamVietnam


Materials and textbooks by regionMaterials and textbooks by region

-0.8

-0.6

-0.4

-0.2

0

0.2

0.4

0.6

0.8

Regions

Sta

nd

ard

ized

sco

res

Materials

Vietnamese books

Math books

Other subjects' books

VietnamVietnam


Type of classroom by school locationType of classroom by school location

0.00

2.00

4.00

6.00

8.00

10.00

12.00

14.00


School location

Nu

mb

er o

f cl

assr

oo

ms

Permanent classrooms

Semi-permanent classrooms

Temporary classrooms

VietnamVietnam


Some benchmarksSome benchmarks

Classroom resourcesClassroom resources (8 items): blackboard, Chalk, (8 items): blackboard, Chalk, teacher table, teacher chair, geometrical instruments, teacher table, teacher chair, geometrical instruments, teachers’ guide (VN&math), and pictures as teaching teachers’ guide (VN&math), and pictures as teaching illustrations. illustrations.

Class sizeClass size: maximum 35 pupils per class.: maximum 35 pupils per class. Total enrolmentTotal enrolment: maximum 1050 pupils per school: maximum 1050 pupils per school BlackboardBlackboard: all classroom should have.: all classroom should have. School resourcesSchool resources (12 items): library, hall, staff room, (12 items): library, hall, staff room,

school head room, store room, first aid kit, medical school head room, store room, first aid kit, medical room, play ground, piped water, fence, music room, play ground, piped water, fence, music

equipment, and sports equipment.equipment, and sports equipment.

VietnamVietnam


Average class size by school locationAverage class size by school locationBenchmark: 35 pupils/classBenchmark: 35 pupils/class

15

20

25

30

35

40


School location

Pu

pils

per

cla

ss

Class size

VietnamVietnam


Percent pupil in full day shiftPercent pupil in full day shiftNational average: 20%National average: 20%

0

5

10

15

20

25

30

35


School location

Per

cent

pup

ils in

full

day

shift

VietnamVietnam


Selected pupil characteristics of Kinh and non-Selected pupil characteristics of Kinh and non-

Kinh pupilsKinh pupils

- 0.8

- 0.6

- 0.4

- 0.2

0

0.2

0.4

0.6

0.8

1

Kinh Non- Kinh

Kinh and non-Kin pupils

Sta

ndar

dize

d sc

ore

Possessions at homeParents' education

Time to help familyGrade repetitionTravel time

Age in month

VietnamVietnam


Socio-economic status and pupil achievementSocio-economic status and pupil achievement

400

420

440

460

480

500

520

540

560

Very disadvantaged Disadvantaged Advantaged Very advantaged

Socio economic status

Sco

res Reading

Math

VietnamVietnam


Means of achievement in reading and mathematics by Means of achievement in reading and mathematics by homework corrected categoryhomework corrected category

410

420

430

440

450

460

470

480

490

500

510

520

Not givenhomework

Never corrected Sometimescorrected

Often corrected Always corrected

Homework corrected

Sco

re Reding

Math

VietnamVietnam


Relationship between years of teaching experience and Relationship between years of teaching experience and pupil reading and mathematics achievementpupil reading and mathematics achievement

0

100

200

300

400

500

600

700

Teacher teaching years

Sco

re PRD500

PMA500

VietnamVietnam


Percentage passed each levelPercentage passed each level

0.0

20.0

40.0

60.0

80.0

100.0

120.0

1 2 3 4 5 6

Levels

Per

cent

age

pass

ed e

ach

leve

l

Math

Reading

VietnamVietnam


Differences in math scores between schools in Differences in math scores between schools in Dong Nai ProvinceDong Nai ProvinceDong Nai rho

0

100

200

300

400

500

600

700

0 10 20 30 40 50 60 70

School

Sco

re m

ath

Series1

VietnamVietnam


Differences in math scores between schools in Differences in math scores between schools in Son La ProvinceSon La Province

0

100

200

300

400

500

600

700

800

0 10 20 30 40 50 60 70

School

Math

500 s

co

re

Series1

VietnamVietnam


The study dataset is stored with the The study dataset is stored with the Ministry of Education and Training, and a Ministry of Education and Training, and a

research report is available in three research report is available in three volumes from the World Bankvolumes from the World Bank

Documents

ASSESSING GRADE 5 PUPIL PERFORMANCE (Reading and Mathematics in 2001) Socialist Republic of Vietnam