PSRF Conference

Dr Ian Goldman Head: Evaluation and Research

28 November 2014

The Presidency Department of Planning, Monitoring and Evaluation

DPME’s role re evidence, including research

The Presidency: Department of Planning, Monitoring and Evaluation

Summary

What is the challenge

DPME’s role re evidence

Evaluations – using research for specific ends

Research – now and way forward

2

The Presidency: Department of Planning, Monitoring and Evaluation

Type of evidence most often used for policy decision-making

(interviews with 55 senior managers, PSPPD, 2012)

3

0 5 10 15 20 25

None

Opinion

Informal

Substantive

Formal Research

Scientific Research

Research Synthesis

All

More of

Current

The Presidency: Department of Planning, Monitoring and Evaluation

The main factors that influence policy (interviews with 55 senior managers, PSPPD, 2012)

4

0 5 10 15 20 25 30

Lobbyists & Pressure Groups

Habits & Traditions

Values

Resources

Judgement

Experience & Expertise

Pragmatics & Contingencies

Evidence

Future

Current

The Presidency: Department of Planning, Monitoring and Evaluation

DPME’s role re evidence

Establishment of PCAS and system of 5 year reviews sought to use evidence

NPC with diagnostic

Range of DPME’s work – outcomes monitoring, MPAT, FSD, evaluations etc seek to use evidence to inform government’s work

Explicit EBPM focus

Project in Presidency on Pro-Poor Policy Development (PSPPD) since 2007 focusing on promoting use of evidence

DPME/PSPPD on training in evidence for DGs/DDGs – run two so far – last October 2014 - 140 applied for this, so running 2 in 2015, as well as range of other initiatives

5

The Presidency: Department of Planning, Monitoring and Evaluation

Operational plan and resourcing

Review, refine and continue

What is known about the

problem

What is the change – desired

and undesired?

Options for addressing the

problem

PLANNING

Policy/Programme planning and

budgeting

OUTPUT

Implementation and monitoring

OUTCOME & IMPACT

DIAGNOSING

Analysis of the problem and

options

Evidence cycle

Theory of change

Design

Implementing the plan

Monitoring the plan, environment and budget

Understand-ing the root causes

Value for money?

Are planned outcomes

being achieved?

Document, evaluate, reflect &

learn

Agenda

Intervention Evaluations Diagnostic

evaluation/research

Smallholder

SR

Repository

Research

Evaluations -

Implementation,

Impact,

Economic

Budgeting?

EPRs

Management

Phakisa/prog

planning

guide

The Presidency: Department of Planning, Monitoring and Evaluation

Evaluations M&E key part of evidence system

Evaluations a form of research, applied to a specific end

39 evaluations completed or underway covering R50 billion of government expenditure

11 selected for 2015/16

16/46 national departments now involved in evaluations, 2 provinces have provincial evaluation plans and 4 more developing them in 2014/15

5 departments have departmental evaluation plans

Systems developed – standards, competences, 18 guidelines, trained >600 staff, QA

Cabinet taking very seriously

Evaluation Repository with >130 evaluations on it as basis for knowledge

7

Stage No

Approved reports 11

Improvement plans

being implemented

8

Served at Cabinet 6

Draft reports 10

Research underway 13

TORs approved 2

Preparation stage 11

Stuck 2

The Presidency: Department of Planning, Monitoring and Evaluation

Evaluations and MTSF

8

Outcome 7 Rural

7.1 Land admin/spatial planning

7.2 Land reform RECAP (2013/14) Restitution (2013/14) Impact of Restitution (2014/15)

7.3 Food security Nutrition for Children Under 5 (2012/13) CRDP (2013/13 CASP (2013/14) Ilima Letsema (2014/15)

7.4 Smallholder farming RECAP (2013/14) Restitution (2013/14) Impact of Restitution (2014/15) CRDP (2013/13 CASP (2013/14) Ilima Letsema (2014/15) Policy evaluation on Smallholders (2014/15) Extension Recovery Programme (2015/16)

7.5 Infrastructure and services

7.6 Non-farm rural enterprises

Outcome 8 Human settlements

8.1 Adequate housing Integrated Residential Development Programme (2012/13) Urban Settlements Development Grant (2012/13) Access to the City (2013/14) Asset Poverty (2013/14) Informal Settlements (2013/14)

8.2 Equitable residential market Social Housing (2014/15)

8.3 Coordination of spatial investment Urban Settlements Development Grant (2012/13)

And

EPRs

The Presidency: Department of Planning, Monitoring and Evaluation

First signs of impacts from evaluations

ECD – new ECD policy been developed including new focus on children from conception (first 1000 days)

Grade R – not having impact in poor schools - moving from emphasis on quantity to quality

Nutrition for Under 5s – increased emphasis, target to reduce stunting from 21-10% in MTSF

Business Process Outsourcing Scheme – scheme redesigned and relaunched

Gov Coordination Systems (clusters) – included in MTSF, developing new guidelines, strengthening secretariats and role of Presidency

Rural (several) – having to rethink and integrate programmes 8 improvement plans being implemented so far

9

The Presidency: Department of Planning, Monitoring and Evaluation

Research to date

Range of research conducted under PCAS, NPC and DPME, especially linked to 5 year reviews and NDP (eg 24 studies commissioned for 20YR)

Supported the NIDS (now going to 4th wave) – longitudinal panel study of 28 000 people where able to track changes in individuals

Set of basic data and indicators Development Indicators – working on data quality

PSPPD supporting use of evidence, notably research, funding set of research studies around social and economic development

Specific research studies, eg on norms and standards, DG time….

Establishing repository of studies in Presidency to complement evaluation repository

10

The Presidency: Department of Planning, Monitoring and Evaluation

Way forward

DPME/PSPPD doing a diagnostic on the research system, what are the systems/skills, how to strengthen, how to ensure it supports the content areas of the NDP/outcomes – how can we influence the system to deliver research related to policy needs. Bringing together Centre of Government departments including NT, DST and DPSA to support the diagnostic and see what configuration may be needed to support this

Introducing innovative tools to support including evidence/gap maps, Rapid Evidence Assessments (REAs), systematic reviews. Piloting these.

Repository of research around outcomes

Supporting departments on development of research (and evaluation) agendas around specific outcomes (already strong re human settlements, rural, economic)

Focusing evaluations on some gaps. Public Sector Research Forum to also suggest areas for evaluations

11

The Presidency: Department of Planning, Monitoring and Evaluation

We welcome the renewed focus on research and evidence.

Thank you

Ian Goldman ian@po-dpme.gov.za

www.thePresidency-dpme.gov.za

12

The Presidency: Department of Planning, Monitoring and Evaluation

Additional slides

13

The Presidency: Department of Planning, Monitoring and Evaluation

NEP 2012/13 Dept Title of evaluation Status as 5 May

dti Implementation/ design evaluation of the

Business Process Services Programme

Final report approved. Report on

way to Cabinet

DBE Impact Evaluation of Grade R Final report approved. Report

been to Cabinet

Health Implementation Evaluation of Nutrition

Programmes addressing Children Under 5

Final report approved

DRDLR Implementation Evaluation of the Land

Reform Recapitalisation and Development

Programme

Final report approved

DRDLR Implementation Evaluation of CRDP Final report approved

DHS Implementation Evaluation of the Integrated

Residential Dev. Programme

Evaluation underway. DHS

procurement has delayed

DHS Implementation Evaluation of the Urban

Settlements Development Grant (USDG)

Evaluation underway. DHS

procurement has delayed

DBE Impact Evaluation of the National School

Nutrition Programme (NSNP)

DBE requested to drop.

Reallocated to 2014/15.

14

Completing Underway About to

start

Delays!

The Presidency: Department of Planning, Monitoring and Evaluation

NEP 2013/14 Dept Title of evaluation Presidency Implementation Evaluation of Government’s Coordination Systems

dti Evaluation of Export Marketing Investment Assistance Incentive programme (EMIAI)

dti Evaluation of Support Programme for Industrial Innovation (SPII)

dti Impact Evaluation of Technology and Human Resources for Industry Programme

(THRIP)

Military

Veterans

Evaluation of Military Veterans Economic Empowerment and Skills Transferability

and Recognition Programme.

DST Evaluation of National Advanced Manufacturing Technology Strategy

SARS Impact Evaluation on Tax Compliance Cost of small businesses

COGTA Impact evaluation of the Community Works Programme (CWP)

DRDLR Evaluation of the Land Restitution Programme

DAFF Impact Evaluation of CASP

DAFF Implementation Evaluation of MAFISA

DHS Baseline for informal settlements targeted for upgrading

DHS Evaluating interventions by DHS to facilitate access to the city.

DHS Diagnostic of whether the provision of state-subsidised housing has addressed

asset poverty for households and local municipalities

DPME Impact Evaluation of the Outcomes Approach

15

Completing

by August Underway About to

start

Delays!

The Presidency: Department of Planning, Monitoring and Evaluation

2014/15

16

Dept Evaluation

DEA Evaluation of the Effectiveness of Environmental Governance in the Mining Sector

(EEGM)

DHET Design Evaluation of the Policy on Community Education and Training Colleges

(PCETC)

DHS Impact Evaluation of the Social Housing Programme (SHP)

DST Evaluation of the Indigenous Knowledge Systems Policy (IKSP)

DSD Diagnostic Evaluation/Programme Audit for Violence Against Women and Children

(AVAWC)

DSD Diagnostic Review of Coordination of the Social Sector Expanded Public Works

Programme

SAPS Economic Evaluation of the incremental investment into the SAPS Forensic Services

DAFF/DRDLR Impact Evaluation of the Ilima Letsema Programme and Irrigation Schemes

DAFF Impact evaluation of MAFISA (quantitative) – through 3ie

DAFF/DRDLR Policy Evaluation of Small Farmer Support

DBE Evaluation of the Funza-Lushaka Bursary Scheme

DBE Impact evaluation of National School Nutrition Programme

DRDLR Impact evaluation of Land Restitution Programme – through 3ie

DPME Impact/implementation evaluation of the MPAT system

DPME Implementation evaluation of the dept strategic planning and APP system

Underway TORs

developed

No TORs

yet

Delays!

The Presidency: Department of Planning, Monitoring and Evaluation

Evaluations selected for 2015/16

Impact/implementation evaluation of the evaluation system (DPME).

Implementation evaluation of citizen-based monitoring programme (DPME).

Agricultural Extension Recovery Plan (DAFF).

Evaluation of the New School Curriculum (DBE).

Evaluation of the Asset Forfeiture Unit Sub-programme (NPA).

Diagnostic evaluation of the Non-Profit Organisations Regulatory Framework and Legislation (DSD).

Implementation Evaluation of the National Drug Master Plan in addressing all forms of Substance abuse (DSD).

Evaluation of the effectiveness of the South African Post-School Education and Training (PSET) quality assurance regime (DHET).

Evaluation of Early Grade Reading in SA (DBE).

Environmental Impact Assessment (submitted for 2014/15 and need to confirm with DEA that they still wish to have the evaluation in 2015/16).

17