Logo

Uganda Evaluation Week 2014Building Evaluation Capacities, Culture and Practices in Uganda

How Evidence Can Be Used for Better Public Services:

Case of Education Sector

Keiichi Ogawa, Ph.D.

Professor Graduate School of International Cooperation Studies

Kobe University 　　　　　　

Outline of the Presentation1. Introduction2. What is Evidence for Policy, 3. Why do we need Evidence?4. Evidence from Uganda’s Education System

-Part I: Basic Statistics (Facts & Figures)-Part II: Effectiveness Analysis on Primary Education in Uganda-Part III: Efficiency Analysis in Lower Secondary Education in Uganda

5. Evidence from International Perspective– Cost-Effectiveness Analysis

2

1. What is Evidence for Policy? (1/2) A discourse or methods that inform the policy

process, rather than aim to directly influence the eventual goals of the policy.

Requires a more rational, rigorous and systematic approach.

Policy which is based on systematic evidence is seen to produce better outcomes.

3

2. What is Evidence for Policy? (2/2/) Better utilization of evidence in policy and practice can

help save lives, reduce poverty and improve development performance in developing countries.

Some examples, Government of Uganda has implemented systematic program of UPE, informed by the results of annual national school censuses and Head Count;

a) Access increased from 3 Million in 1996 to 8.4 Million 2010;b) No. of schools increased from 12,500 in 2000 to 19,797 in 2009c) GER reduced from 128% in 2000 to 114% in 2010;d) Teachers doubled (74,000 in 1995 to 145,302 in 2009)

4

3. Why do we need Evidence?Shaxson (2005) argues that we need evidence to: Understand the policy environment and how it’s changing;

Appraise the likely effects of policy changes, choice to different policy options and assessing their impacts;

Links between strategic direction, intended outcomes and policy objectives;

Determine what we need to do to meet our strategic goals or intermediate objectives;

Influence others so that they help us achieve our policy goals and take them through to delivery;

Communicate the quality (breadth and depth) of our evidence base to meet the government agenda.

5

EVIDENCE FROM UGANDA’S

EDUCATION SYSTEM

6

A) Background In the past two decades, Uganda embarked on major education reforms

(e.g. UPE in 1997 and USE in 2007) which have largely focused on equalizing educational opportunities as part of a broad development strategy to achieve EFA and MDGs.

Significant investments have been made towards inputs in the areas of: Curricula, teaching/ learning processes, school inspection and support supervision teachers’ accommodation, absenteeism, school facilities and infrastructure. These have paid off with an increase in enrolment figures over time. The challenge however that still stands is whether the millions of children in school for whom these investments have been made are learning.

Education continues to take a large share of the budget but is also a sub-sector where the evidence of waste is evident and rampant (MoFPED 2012). One key area of current debate has been inability and poor quality of school Management to use the available resources efficiently (MoES 2010).

7

8

B) Structure of Uganda’s Education SystemThe Structure of Uganda’s Education System

Source: Developed by Author (2010)

Technical, Business and Teacher colleges /schools

Pre-Primary (2 years)

Primary (7 years)

Upper-Secondary (2 years)

Lower-Secondary (4 years)

University Degree /Diploma

Certificate/Diploma

3/5 years

2/3 years

Duration Awards Professional Academic

PART 1:

BASIC STATISTICS – FACT & FIGURES

9

A) Evidence from UPE Program – Access Issues

Trends in Primary School Enrollment by Gender

10

Source: MoES (2010)

B) Evidence from UPE Program – Quality Issues

Percent of P3 and P6 pupils rated proficient in Literacy

11

2003 2004 2005 2006 2007 2008 2009 2010

Total (P.3) 34.3% 36.7% 39% 45.6% 45.5% 44.5% 55.9% 57.6%

Male (P.3) 33.1% 35.1% 37% 44.2% 43.8% 43.8% 55.2% 57.9%

Female (P.3) 35.5% 37.8% 40% 46.9% 47.2% 45.3% 56.5% 57.3%

Total (P.6) 20% 25.0% 30% 33.5% 49.6% 47.9% 48.1% 50.2%

Male (P.6) 20.3% 26.2% 32% 33.4% 48.2% 47.9% 47.9% 49.7%

Female (P.6) 19.5% 23.8% 28% 33.6% 50.8% 47.8% 48.2% 50.7%

0.0%

10.0%

20.0%

30.0%

40.0%

50.0%

60.0%

70.0%

%ag

e of

Pup

ils

Year

Total (P.3) Male (P.3) Female (P.3) Total (P.6) Male (P.6) Female (P.6)

Source: UNEB Surveys, (2003-2010)

C) Evidence from UPE Program – Quality Issues

Percent of P3 and P6 pupils rated proficient in Numeracy

12

2003 2004 2005 2006 2007 2008 2009 2010

Total (P.3) 42.9% 44.0% 45.0% 42.6% 44.8% 71.4% 71.3% 72.8%

Male (P.3) 43.9% 45.0% 46.0% 45.4% 46.3% 74.6% 72.8% 74.1%

Female (P.3) 41.9% 43.0% 44.0% 39.6% 43.3% 68.1% 69.7% 71.6%

Total (P.6) 20.5% 26.8% 33.0% 30.5% 41.4% 53.5% 53.3% 54.8%

Male (P.6) 25.7% 32.4% 39.0% 34.4% 45.9% 58.8% 58.7% 57.9%

Female (P.6) 15.3% 21.2% 27.0% 26.7% 37.2% 48.4% 48.1% 52.1%

0.0%

10.0%

20.0%

30.0%

40.0%

50.0%

60.0%

70.0%

80.0%

%ag

e of

Pup

ils

Year

Total (P.3) Male (P.3) Female (P.3) Total (P.6) Male (P.6) Female (P.6)

Source: UNEB Surveys, (2003-2010)

D) Evidence from UPE Program – Internal Efficiency

Progression Rates between Classes for 1997-2004 Cohorts

13

P2 P.3 P4 P5 P6 P7

1997 61% 52% 45% 39% 33% 22%

1998 73% 68% 62% 55% 46% 29%

1999 72% 70% 66% 60% 47% 30%

2000 71% 71% 67% 56% 46% 29%

2001 71% 69% 61% 54% 45% 28%

2002 67% 62% 55% 50% 41% 28%

2003 62% 61% 55% 49% 42% 29%

2004 64% 65% 60% 54% 45% 30%

0%

10%

20%

30%

40%

50%

60%

70%

80%

Prog

ress

ion

Rate

Class

1997 1998 1999 2000 2001 2002 2003 2004

Source: MOES (2010)

14

E) Uganda Compared to SSA Countries - SACMEQPupils’ mean reading scores per country

0 100 200 300 400 500 600 700

Malawi

Zambia

Namibia

Lesotho

Zanzibar

Uganda

S.Africa

SACMEQ

Zimbabwe

Mozambique

Botswana

Swaziland

Mauritius

Tanzania

Kenya

Seychelles

Pupil's Mean Score

SACM

EQ C

ount

ry

2000' 2007'

Source: Drawn by Author Based on SACMEQ 2000 and 2007 Surveys

15

F) Compared to SSA Countries - SACMEQPupils’ mean mathematics scores per country

0 100 200 300 400 500 600 700

Namibia

Malawi

Zambia

Lesotho

Zanzibar

S.Africa

SACMEQ

Uganda

Botswana

Swaziland

Tanzania

Mozambique

Seychelles

Kenya

Mauritius

Zimbabwe

Pupil's Mean Score

SACM

EQ C

ount

ry

2000' 2007'

Source: Drawn by Author based on SACMEQ 2000 and 2007 Surveys

G) Evidence from USE Program –Access Issues

USE Enrollment trends over the years, 2007-2011

16

Source: Created by the author based on data from MoES (2011)

H) Evidence from USE Program –Quality Issues

Completion rate at S4

17

Source: Created by the author based on data from MOES (2011)

I) Evidence from USE Program –Quality Issues

Transition rates to S5

18

Source: Created by the author based on data from MOES (2011)

Part IIEffectiveness Analysis on

Primary Education in Uganda

19

A) Research Issues/QuestionsTo what extent does the family factors (e.g. pupil age & sex, language spoken at home, pupil staying with parents, ensuring pupil’s daily attendance, the number of books at home, pupil helped with homework at home, parents education, home-possessions and having electricity etc.) correlate with pupil’s academic achievement?

To what extent do the school factors (e.g. subject teachers appreciating the conditions of classroom, teachers qualification and years of experience, a pupil repeating grade six, pupil borrowing books from the library, pupil given home work, school head and teacher’s qualifications, number of years of service (experience), school type influence pupils’ academic achievement?

To what extent does the community variables (e.g. community’s regular meetings with the teachers and school heads, paying salary for additional staff and topping-up teachers’ salaries, location (rural and urban) of the school influence on the pupil’s academic achievements?

20

B) Methods and DataUse an Education Production function as proposed by Glewwe and Kremer (2005)

Where Tis is the score of student i from school s,Hj = is a vector of home context characteristics,Sk =is a vector of school system context characteristics,Cl= is a vector of community characteristics

Data IssuesSouthern African Consortium for monitoring Education Quality (SAQMEQ II & III). It focused on Uganda because it is one of the fifteen countries that participated in the SACMEQ survey.

21

C) Key Findings

22

Reading Mathematics Predictor variables Coefficient P-value Coefficient P-value

Home related variables Age of the pupil (<=11 years) +ve *** +ve *** Age of the pupil (12years) +ve *** +ve ** Age of the pupil (13 years) +ve *** +ve ** Age of the pupil (=>14 years): reference category Pupil sex (Male=1) +ve *** +ve *** Pupil speak English at home (yes=1) +ve *** +ve *** Pupil stay at home with parents (yes=1) +ve ** Secondary education of mother +ve *** +ve ** Tertiary education of mother +ve +ve Secondary education of father +ve -ve * Tertiary education of father +ve +ve Having electricity at home (yes=1) +ve *** +ve ** Total possessions at home +ve *** +ve *** Pupil helped with home work at home (yes=1) +ve * +ve *** Number of days pupil is absent -ve *** -ve ***

School factors School type (government =1) -ve *** -ve *** Pupil repeating grade six (yes=1) -ve *** -ve *** Tertiary education of school head +ve *** +ve *** Upper education of class teacher +ve *** +ve Sex of school head (Male=1) -ve ** Years of service of the head teacher +ve *** +ve ** School resources +ve *** +ve *** Conditions of classrooms are good (yes=1) +ve +ve *** Days in a month class teacher has been absent -ve *** -ve *** Pupil given homework at school (yes=1) +ve *** +ve

Community factors Community meeting teachers at school – yearly +ve *** +ve *** Community meeting teachers at school – termly +ve *** +ve *** Community meeting teachers at school – monthly +ve *** +ve *** School location (rural=1) -ve *** -ve *** Payment of salaries of additional teachers +ve *** +ve * Paying teachers top-ups of the normal salary +ve *** +ve *** Constant +ve *** +ve *** N 5227 5221 Adjusted R-squared 26.9% 18.3% Asterisks denote significance level; *** = 1percent, ** = 5 percent, * = 10 percent

D) Implications for PolicyThe community meeting teacher variable emerges as a significant variable explaining performance. Government should organize monthly or quarterly sensitization meetings with communities so that they know the value of education to children.

Rural parents may need extra form of financial support to develop schools and personal income. Central government should increase the equalization funds currently being paid to disadvantaged local governments and support schools started by communities

The ministry of education and sports could find ways of dealing with grade repetition without lowering the standards of achievement by employing (Brophy 2006) strategies such as – early intervention, collaboration with parents, and supplementary instructions.

Central government should develop guidelines and put in place a policy for all local governments and school head teachers to regularly sensitize parents to provide the basic needs of their pupils and provide help for them at home with their studies.

23

24

Part III

Efficiency Analysis in Lower Secondary Education in Uganda

A) Research Issues/Questions

Question 1: What is the current technical efficiency level of schools in Uganda?

Question 2: How do school factors (e.g. school size, proportion of female students, heterogeneity of students body, number of career guidance services and number of female stances, school ownership, USE status, co-educational type and boarding type & location) affect its technical efficiency?

Question 3: How do students and family socio-economic (e.g. student’s age, per student family expenditure on education and population density) factors affect school technical efficiency?

25

B) Methods & DataObjective 1: Estimating School Technical Efficiency indices using non-

parametric technique - Data Envelopment Analysis (DEA) Model

Objective 2-3: Use of Tobit Model

are efficiency scores from DEA analysis & εi is the error term school size (SIZ), boarding status (SBT), co-educational (SCT), proportion of female students (PFS), female stance ratio (FSR), heterogeneity of students’ body (HET),career guidance activities (CGA), school USE status (USE), ownership (OWN), location (LOC), students’ age (AGE), family expense on education (HHE), population density (POP) and regional dummies for eastern (EDV), northern (NDV) and western (MDV); Βj are vectors of parameters to be estimated

26

)1,ˆ(min&),0(~

)()()()()()()()()(

)()()()()()()(ˆ

2

1615

141312111098

76543210

SSN

MDVNDVEDVPOPHHEAGELOCOWNUSE

CGAHETFSRPFSSCTSBTSIZS

i

i

i

C) Key Results – Objective 1

27

Distribution of Efficient and Inefficient scores (Analysis based on VRS) Efficient schools (n=70) Inefficient schools (n=213) All (n=283) Mann-Whitney

test N0. % of

the group Mean N0. % of

the group Mean N0. % Mean Z-Statistic (P-

value) Location

Urban 18 18.0 1.0 82 82.0 0.83 100 100 0.86 -0.348 (0.727) Rural 52 28.4 1.0 131 71.6 0.79 183 100 0.85

Ownership Government 49 24.5 1.0 151 75.5 0.81 200 100 0.85 -0.211

(0.833) Private 21 25.3 1.0 62 74.7 0.79 83 100 0.84 USE status

USE 38 20.4 1.0 148 79.6 0.79 186 100 0.83 4.528 (0.000) Non-USE 32 33.0 1.0 65 67.0 0.84 97 100 0.89

All 70 24.7 1.0 213 75.3 0.80 283 100 0.85 Source: Created by the author (2012)

Summary of Results – Objective 1 Of 283 schools, 70 (24.7%) of them are efficient, and the rest (75.3%) are

far from the efficiency frontier.

52 (28.4%) out of 183 are efficient in rural compared to 18 (18.0%) of the 100 urban schools.

Of 200 public schools, 49 (24.5%) of them are efficient compared to only 21 (25.3%) out of 83 private schools.

38 (20.4%) of the 186 USE are efficient compared to 32 (33.0%) of the 97 non-USE schools.

There is significant difference (at 1%) between productive efficiencies of schools under USE policy and non-USE

28

D) Key Results – Objectives 2 & 3

29

Dependent variable: Technical Efficiency Scores OLS Tobit Quantile regressions 10th 30th 50th Students’ age (>16.9 years) -ve *** -ve *** -ve *** -ve *** -ve *** >15.9-16.9 years -ve *** -ve *** -ve *** -ve *** -ve *** >14.9-15.9 years -ve *** -ve *** -ve *** -ve *** -ve ** <=14.9 years (RC) Family expenditure on education +ve *** +ve *** +ve * +ve ** +ve Population density (Av. family size) -ve -ve +ve +ve +ve School size (000’s) +ve *** +ve *** +ve +ve *** +ve ** School size-squared -ve *** -ve *** -ve ** -ve *** -ve ** Location (Urban=1) -ve -ve -ve -ve -ve Ownership (Public=1) +ve +ve +ve -ve -ve USE status (USE=1) -ve -ve -ve +ve +ve School boarding type (Yes=1) +ve +ve +ve +ve +ve School type (co-educational=1) +ve +ve ** -ve * -ve +ve Proportion of female students +ve *** +ve *** -ve -ve +ve Female latrine stance ratio +ve ** +ve *** +ve +ve * +ve Heterogeneity of student body -ve -ve ** -ve ** -ve -ve Number of career guidance services +ve * +ve ** +ve +ve +ve Eastern region +ve ** +ve * +ve ** +ve * -ve Northern region +ve *** +ve *** +ve *** +ve *** +ve Western region +ve *** +ve *** +ve *** +ve *** +ve Central region (RC) Constant +ve *** +ve *** +ve *** +ve *** +ve *** N 283 283 283 283 283 Pseudo R2/ Adj. R-squared 29.7% 28.4% 31.9% 25.5% 26.1% Source: Created by the author (2012) Note: Asterisk denote significance level, *** = 1%, **= 5% and *= 10%; Standard errors are in parentheses

Summary of Results – Objectives 2& 3 School factors that demonstrate significant; Positive effect :- School size, - Co-educational status, - Proportion of female students and their stances,- Number of carrier guidance services.- Location (regional) effects

Negative effect :- Heterogeneity of students body

Student & Family factors: Positive effect: Family expenditure, regional

effects; Negative effect: Students’ age group effects

30

E) Implications for Policy

The introduction of USE policy increased schools’ enrolments that constrained resources creating harsh conditions for schools to operate. It was/is considered “transition” period engulfed with management and financing challenges.

Expenditures on education reflect increased investment in students’ learning and also financial mark-up to schools. So, government may design funding mechanism to public schools to narrow the financial economic differences between private and public schools.

Most schools attract children from poor families and charge low fees. Funding constraints suffocates other basics.

In large schools, high achievement are possible as long as resources are available and proper use of them is assured and sustainable. The education policies should emphasize improving school size as long as class sizes are maintained standard/reasonable.

31

EVIDENCE FROM INTERNATIONAL PERSPECTIVE

COST-EFFECTIVENESS ANALYSIS:

32

Approximate cost for increasing years of schooling by 1 year

Source: Ogawa, Nakamuro, & Hoshino (2009)

6. 　 Cost-Effectiveness Analysis of Education ProjectCost-Effectiveness Analysis of Education Project Evaluation with Experimental Data: An International Comparison

(Upper axis)

(Upper axis)

(Upper axis)

(U.S. dollars)

(U.S. dollars)

Microfinance (Guatemala)

Cash transfer (Mexico)

Education voucher (Colombia)

Scholarship (Kenya)

School construction (Indonesia)

Increase in provision of assistant teachers (India)

Introduction of teacher incentives (India)

School feeding (Bangladesh)

Deworming drugs (India)

School feeding (Kenya)

Free of recurrent education costs (Kenya)

Deworming drugs (Kenya)

Approximate cost for increasing a test score by 0.1 standard deviation

Source: Ogawa, Nakamuro, & Hoshino (2009)

6. 　 Cost-Effectiveness Analysis of Education ProjectCost-Effectiveness Analysis of Education Project Evaluation with Experimental Data: An International Comparison

(U.S. dollars)

Education voucher (Colombia)Reducing class-size (Honduras)

Early childhood education program (Philippines)Computer-aided instruction (India)

Textbook (Kenya)Teacher incentives (India)

Teacher incentives (Kenya)Scholarship (Kenya)

Classroom renovation (Ghana)Capitation grant (Uganda)

School furniture (Philippines)Supplementary lesson implementation (India)

Teacher training (Honduras)Workbook for students (Philippines)

Black board (Ghana)

Thank You