M&E, STATISTICAL & INFORMATION SYSTEMS

LUSAKA /ZAMBIA 16 May 2013

By Ledule M Bosch Chief Director: Monitoring & Evaluation

Department of Public Service & Administration South Africa

2

Presentation Overview

• Introduction: M&E

• Why Monitor?

• What Monitoring is not

• Evaluation Principles

• What Evaluation is not

• Indicators & Pitfalls to avoid

• Why M&E Information Systems

• Benefits

• Concluding Remarks

o

Sections of the Constitution that deals with services rendered by each government department/Institution

Country’s Vision (e.g. 2030 Vision) Medium Term Strategic Frameworks; Poverty Reduction Strategies / National Development

Plans Legal mandate (Regulations, prescripts etc) Policies &Programs International / Continental Obligations (e.g MDG’s)

4

Why Monitor?

o - Prevent Project Failures

o - Prevent cost and time over-runs

o - Keep a track of progress of Project implementation

o - Ensure that resources are expended as planned

o - Above all, it can ensure that the benefits of government reach the target group, citizens and non-citizens

5

What Monitoring is not…

oReporting ≠ Monitoring

oInspection ≠ Monitoring

oSupervision ≠ Monitoring

oAudit* ≠ Monitoring

oSurveillance ≠ Monitoring

oReview ≠ Monitoring

6

What then is Monitoring?

Monitoring is a

o specialized,

o systematic,

o dynamic, and

o semi-autonomous management tool to ensure that government serves the target group- Citizens and Non-Citizens- taking into account the interests of various stakeholders and the emerging challenges facing government

7

What is not Evaluation

Analysis ≠ Evaluation

Measurement ≠ Evaluation

Assessment ≠ Evaluation

Appraisal ≠ Evaluation

Audit ≠ Evaluation

Monitoring ≠ Evaluation

8

EVALUATION PRINCIPLES

Accountability

Transparency

Flexibility

Integrity

High professional and ethical standards

Independence

Interdependence

Partnership

Objectivity

Know your Stakeholders & their Information Needs :Example of Govt Department

Development Partners/

Donors

Parliament/Portfolio Committees/Oversight

Institutions

International/

Continetal Bodies

Govt.

Department

The Public

Transversal Departments

Academia Research

Institutions

Service Delivery Departments

9

o One key tool in measuring outputs and outcomes is the indicator;

o In effect, this is a measuring stick of development programs;

o In the past, government may have been satisfied to know that their funds have been “put in” in certain program areas;

o Today the situation is quite different. Government want to know what came “out” from the funding ;

o The focus has shifted from inputs to outputs and from outputs to outcomes and impacts

Understanding the Rationale for

Indicators & information Systems

10

Dead Cockroach

INDICATORS & PITFALLS TO AVOID

Example: “The Ants and the Dead Cockroach” Imagine this

scene

11

INDICATORS & PITFALLS TO AVOID…

“The Ants and the Dead

Cockroach”

Imagine this scene

•Thousands of ants working to bring the dead cockroach towards their nest. (Teamwork); •Focus on any individual ant and you may find that it may be doing number of things (key player, pulling the dead cockroach in the right direction, towards the nest; it may unwittingly be one of the minority pulling in the wrong direction; it may be running around doing nothing, but very excited to be near the action

SO!

How do you measure this ants contribution to the task at hand? If we ask it, it will say it contributed actively, as it indeed shares the dream of having the cockroach at the door of their nest. It can also tell us how many minutes (itself) was pulling at the cockroach, but in which direction, it might not be able to tell us

12

INDICATORS & PITFALLS TO AVOID…

“The Ants and the Dead Cockroach” Imagine this

scene

•What else can we measure? We can go to a higher level of assessment. E.g We can measure the average speed of progress, but this will be affected by factors such as weather, the time of the day etc. •Or we can assess whether the objective was met- did the cockroach arrive at the ants nest door? The moral of this story is that balance has to be maintained between the time and effort spent doing what needs to be done, and that spent reporting what you have done

13

So why do we store, update on all of this

indicator data collected?

• To produce standardized and accurate monthly, quarterly, bi-annual and annual statistical reports, fact sheets and templates;

• To ensure the quality of data and reports;

• To set up a national database of key monitoring and evaluation indicators;

• To simplify and facilitate the task of sharing vital information needed for decision making;

• To share data sets with strategic partners;

• To promote sound research 14

WHY M&E STATISTICAL INFORMATION

SYSTEMS?

• Storage of information on key indicators of performance and service delivery;

• Production of accurate scientific reports needed for planning and improvement of services;

• Efficient production of standardized reports;

• Identification of cost cutting measures;

• Promotion of sound M&E research;

• Standardized and efficient reporting on activities, outputs & outcomes

• Ability to perform forecasting and trend analysis in future

• Ability to share vital information with other local, development partners/Donors and international institutions

15

Developmen

t of a

comprehens

ive M&E

Framework

Define key

informatio

n needs &

requireme

nts

Develop/En

hance

reporting

templates

Define/R

efine /

standard

ize Core

KPIs to

measure

perform

ance

Set up a

centraliz

ed

database

for M&E

to store

past &

present

datasets

Develop a

consolidate

d matrix of

M&E

indicators

needed for

sound

planning &

decision

making

Dissemi

nate

reports

and

findings

1 2 3 4 5 6 7 9 10 8

Building an M&E System : Key steps Produce

standardized

monthly, quarterly,

bi-annual and

annual reports,

fact sheets &

templates

M&E for

data

analysis

and

baseline

Analyze

past and

present

M&E data

sets

systemati

cally

16

Monitoring & Evaluation Chain of Delivery

Government Priorities/ MTSF/MTEF Poverty Reduction Strategies/National Development Plans

Strategic/ Annual Performance Plans

Tools Series

Framework for M&E

Indicator Matrix

Manuals & Guidelines (M&E Tools)

M&E System

Monitoring System Monitoring Plans

Multi Year Evaluation Plans Evaluation System

Monitoring & Evaluation Reviews &

Assessments

Dissemination of M&E Findings/Information Website, Publications, Newsletter, M&E Portal

Feedback loop (Policy) 17

Data Collection Primary/Secondary Data

Obtain Baseline Data

Development/Review of

Objectives and Indicators

Evaluation / Evaluability

Assessment

Self – Assessment Tools & Early Warning System

(PMW) & DPSA MPAT Indicators

Evaluations: Formative &

Summative (Surveys Service

User/Employee

Monthly/ Quarterly/ Bi-annual / Annual M&E Reports (PMW /HR Strategic Reports, etc)

Service User/

Management Information System ( Compliance Monitoring System, M&E

Portal etc

M&E Capacity Development Stakeholder Information Needs Analysis

/M&E / International Learning Networks & Benchmarks

Key M&E Outputs in the Delivery Chain

18

19

AN INTEGRATED M&E SYSTEM: EVALUATION

Use of primary/

Secondary data

Evaluation Assessment/ Evaluability

Assessment

Obtain Baseline Data

Mid Term/ Final/ Comprehensive / Thematic/ Strategic Evaluations

Findings

Use of information for efficient programme administration

Use of information to improve quality of service provision

Dissemination of information for institutional learning and transparency

Recommendations Solutions

Framework

Evaluation

Information System

EXECUTING AGENCY

EXAMPLE : FRAMEWORK FOR M&E EARLY WARNING

SYSTEM S

YS

TE

M

20

OVERVIEW OF SYSTEM

VALUE

SYSTEM

RESPONSE

?

21

Provide ‘management information’ that can be used to

assess organisational performance.

Facilitate review of performance over a period of time

and assess progress on performance changes and

reforms introduced.

Determine the specific interventions required to ensure

that the public service and specific public service

organisations function optimally.

Establish performance benchmarks and achieve

excellence through comparative assessments within the

public service.

Facilitate learning from data and information, and

enhance knowledge exchange between public service

organisations.

Value OBJECTIVES OF HE SYSTEM

Provide ‘management information’ that can be used

to assess organisational performance.

Facilitate review of performance over a period of

time and assess progress on performance changes

and reforms introduced.

Determine the specific interventions required to

ensure that the public service and specific public

service organisations function optimally.

Establish performance benchmarks and achieve

excellence through comparative assessments

within the public service.

Facilitate learning from data and information, and

enhance knowledge exchange between public

service organisations.

VA

LU

E

? 22

AG AUDITOR-GENERALS

REPORTS

PERSAL PERSONAL AND SALARY

ADMINISTRATION SYSTEM

IYM IN-YEAR-MONITORING

REPORTS

Value COLLATION APPROACH V

ALU

E

INDICATORS

RANGE

WEIGHT

23

o This presentation embodies what elements of M&E System needs to be taken into account way before indicators can be selected and a system can be designed and implemented;

o Continuous Training & capacity development on all aspects on indicator, target setting, baseline collection, data collection, collation and analysis is critical to sustain an M&E System

Concluding Remarks

24

THANK YOU