Evaluering – oppskrift på TOR

Evaluering – oppskrift på TOR

Anette P. Simonsen24. oktober 2012

9:00-16:[email protected]

Bistandstorgetw

w w.

R e

s u

l t a

t s t

y r

i n g

. n

o

INTROEvaluering som en del av M&E

w w

w. R

e s

u l t

a t

s t y

r i n

g .

n o

Monitoring EvaluationWHO? Internal, involving project

implementers Can be internal or external to organisation

WHEN? OngoingDuring project

OngoingPeriodic and after project

WHY • Check progress, • inform project

decisions and remedial action

• update project plans• support accountability

• Assess progress and worth, • identify lessons and

recommendations for longer term planning and organisational learning

• provide accountability

Link to LFA

Focus on activities, outputs and short term outcomes

Focuses on outcomes and overall goal.

Monitoring and Evaluation

M

ål- o

g re

sulta

tsty

ring

Ane

tte P.

Sim

onse

n

Monitoring:• refers to the regular follow-up and assessments during the

implementation of a project or programme. • normally provides information on expenditure, activities and

outputs and a certain extent outcome • is used for adjusting the course of the project/programme as

well as writing progress reports. Evaluation • is the systematic and independent assessment of a project/

program that is performed at certain stages of the project (e.g. mid-term, end of project).

• would provide information on output, outcome and impact• is used for adjusting the course or assessing continuation or

ending a project.

Definitions M&E

M

ål- o

g re

sulta

tsty

ring

Ane

tte P.

Sim

onse

n

M&E definition again..

Evaluation approaches often have high methodological standards, a separate budget and often scientific requirements for measurements and analysis. Monitoring on the other hand is a routinetask, regularly ‘checking up on how things are’ and recording this information in a way that is useful both

w w

w. R

e s

u l t

a t

s t y

r i n

g .

n o

Observing Change N-P-Aid

http://www.norad.no/no/tilskudd/s%C3%B8k-st%C3%B8tte/_attachment/153487?=true&_ts=12fd9e2bf57 Se spesielt side 3:

«Beslutning om støtte skal tas etter en helhetsbedømmelse av søknaden. Hovedsakelig er kriteriene delt i to; de som gjelder for vurdering av organisasjonen og de som gjelder vurdering avbistandsvirksomheten. Sentrale punkter i vurderingen av organisasjon er:• Kapasitet og kompetanse til å nå organisasjonens mål• Organisasjonens kostnadseffektivitet, det vil si evnen til å

gjennomføre organisasjonens mål med lavest mulige kostnader.• Systemer for resultatoppfølging• Evne til å identifisere og håndtere risiko

Norad Regelverk for støtte til sivilsamfunnsaktører

w w

w. R

e s

u l t

a t

s t y

r i n

g .

n o

http://www.norad.no/no/tilskudd/s%C3%B8k-st%C3%B8tte/_attachment/153487?=true&_ts=12fd9e2bf57



§ 16 Evalueringer• Alle virksomheter skal sørge for at det gjennomføres

evalueringer for å få informasjon om effektivitet, måloppnåelse og resultater innenfor hele eller deler av virksomhetens ansvarsområde og aktiviteter. Evalueringene skal belyse hensiktsmessighet av eksempelvis eierskap, organisering og virkemidler, herunder tilskuddsordninger.

• Frekvens og omfang av evalueringene skal bestemmes ut fra virksomhetens egenart, risiko og vesentlighet.

Økonomireglementet sier: w

w w.

R e

s u

l t a

t s t

y r

i n g

. n

o

M&E har 3 formålM&E skal bidra til: • Kontroll av virksomheten - Forsikre interessenter om at midlene

blir brukt på riktig måte, for avtalte mål og at de ulike samarbeidspartene oppfyller sine forpliktelser

• til beslutningsprosesser – gi informasjon som underbygger de valg organisasjonene står overfor, når de må fatte beslutninger om prosjekter/ programmer skal fortsette, avsluttes eller endres

• til å fremme organisatorisk læring – ved å gi aktuelle aktører kunnskap og informasjon om prosjekter/ programmer samt den sammenhengen de virker inn i. Ingen begrensning i hva som er mulig læring.

w w

w. R

e s

u l t

a t

s t y

r i n

g .

n o

Kruse & Forss (2000)

Look back at:• Does the project/programme have a clear goal

hierarchy and a set of M&E plans?• Have the plans been monitored?• Are there good and regular monitoring reports?• Are the project staff available /still contactable?• Are the stakeholders involved? • Are the donor(s) involved?

Preparing for Evaluation

From Local Livelyhoods

w w

w. R

e s

u l t

a t

s t y

r i n

g .

n o

Action Plan for Evaluation- Approving the activity and the budget

w w

w. R

e s

u l t

a t

s t y

r i n

g .

n o

Action Plan versus TOR

The Action plan• Formulates the internal “project” of performing

an evaluation • It documents the approval of the decision to

perform an evaluation and the budget for it. The TOR • is a work description for the team performing

the evaluation• It constitutes only one part of the Action Planw

w w.

R e

s u

l t a

t s t

y r

i n g

. n

o

Action Plan for Evaluation1. Background and justification

– Need for evaluation at this point– Previous relevant evaluations / reviews

2. Who will be involved in evaluation / cooperation with other stakeholders/donors on the evaluation

3. Target group (audience)4. Objective of evaluation5. Any special requirements to team composition, method etc.?6. Outputs/main activities

– Planning phase– Implementation– Follow up

7. Risk and uncertainties8. Organisation and responsibilities9. Budget

w w

w. R

e s

u l t

a t

s t y

r i n

g .

n o

Donor

DonorDonor Donor

Justification for evaluationw

w w.

R e

s u

l t a

t s t

y r

i n g

. n

o

Stake-holder

Norwegian NGO

Stake-holder

Stake-holders

Stake-holders

PartnerTanzania

ProjectIMPLEMENTATION LEVEL

1. Background and Justification

Do we really need an evaluation at this point?What is the problem we are trying to solve?E.g.• The M&E system is weak so we need to increase

level of information on progress and or achievement of results?

• The target group is complaining on quality of services?

• We need to secure funding by a more objective voice on achievement of results?

w w

w. R

e s

u l t

a t

s t y

r i n

g .

n o

Initial Assessment

Logframe & Indicators

M&E Planning

Baseline Study

Midterm Evaluation

and/orReviews

Final Evaluation

(endlinesurvey)

Disseminationand Use of

Lessons

FIGURE X: Key M&E activities in the Project Cycle*

OngoingReportingReflectionLearning

Planning

Monitoring

Evaluation

Project Start

Project Follow-up

Project End

M&E in project Cycle

w w

w. R

e s

u l t

a t

s t y

r i n

g .

n o

When? Reporting to donors

Program year Deadline and type of Report

Reviews and evaluations

2009 May 2010 –Progress report

2010 May 2011 - Progress report

2011 May 2012 - Progress report

2012 May 2013 - Periodic Results Report

2013 May 2014 –Final Reportw w

w. R

e s

u l t

a t

s t y

r i n

g .

n o

Example from 5 year programme:

Why? Status report Norad

• 3.1 Describe the reviews/evaluations that have been carried out during the year

• In what way(s) did the review/evaluation contribute to learning and improvement of the programmes?

• Are external reviews submitted to Norad’s data base for evaluations?(state type of evaluation – external, internal or mix external/internal)

w w

w. R

e s

u l t

a t

s t y

r i n

g .

n o

Timing of evaluationsAppraisal/ evaluation – before start-upMid-Term Evaluation – towards middle of programme periodEnd-Term evaluation – at the end of programmeFormative evaluations – during programmeSummative evaluations – at the endEx-post evaluations – some time afterImpact evaluations – at the end – some time after

w w

w. R

e s

u l t

a t

s t y

r i n

g .

n o

OECD evaluation on DAC principles 1998:

• Evaluations most useful when they are linked to current and future programmes rather than historical perspectives (although accountability requires the latter type)

w w

w. R

e s

u l t

a t

s t y

r i n

g .

n o

2. Who will be involved in evaluation / coop with other stakeholders

Discuss evaluation plans and design with the stakeholders (staff, partners, benficiaries, others) and include the primary target group.• Who in the commissioning organisation is

familiar with project/programme• Who are the stakeholders?• Which other donors are involved in

Project/programme or even thematic area og geographical area.

• Others?

w w

w. R

e s

u l t

a t

s t y

r i n

g .

n o

3. The target group (audience)

Who will read, learn from and use the evaluation report should be one of the a determining questions in developing the terms of reference and designing the evaluation. In other words who is the primary target group that should be able to make use of the report? Should e.g. Norwegian NGO / the partner /stakeholders / members / taxpayers/ donors make use of the evaluation?

w w

w. R

e s

u l t

a t

s t y

r i n

g .

n o

OECD evaluation on DAC principles 1998:

• Evaluations are performed mostly because donors ask for it. Little attention is paid to the demand, the use and usability of evaluations, and differentiation according to different audiences

w w

w. R

e s

u l t

a t

s t y

r i n

g .

n o

4. Objective of the evaluation

The purpose of evaluations is normally a combination of a need for control (accountability) to contribute to decision making and for learning• Partner – all 3• Norwegian all 3 • Donor…. Who cares?

w w

w. R

e s

u l t

a t

s t y

r i n

g .

n o

Objective of Evaluation

• The implementing partners will have increased knowledge and lessons learned in their continuous work within the thematic areas in their respective countries.

• The Norwegian NGO will be better equipped for future disasters of similar order of magnitude, both in the initial and recovery phase.

• The private and corporate donors and the Ministry of Foreign Affairs will gain knowledge about the results of their donations.w w

w. R

e s

u l t

a t

s t y

r i n

g .

n o

Who is the target group here?

Objective of Base line study

The main objectives of the baseline survey are:• To get an initial baseline standing/situation/ context of

the current and existing governance and livelihood situation, systems and structures in the impact areas

• To establish baseline indicators for the programme that will imply a clear monitoring and evaluation plan (M&E) and platform for impact evaluation (IE)

• To source information that is required to describe qualitatively and quantitatively the outcome indicators and output indicators for the Caritas Malawi programme on governance and livelihood.w w

w. R

e s

u l t

a t

s t y

r i n

g .

n o



• Have conversations on the internal functions, including financial management, in each organisation involved in the visit in order to strengthen the organisation

• Have conversations about the cooperation (communication, interaction) between the organisations in order to improve cooperation

• The review should be summarized in a report in which the strengths and weaknesses are described, and proposed changes in procedures and communication within and between organisations are described.

w w

w. R

e s

u l t

a t

s t y

r i n

g .

n o



• Conducting a mid-term external review in 2011 will establish an overall picture of whether the five-year programme is on track towards 2013 and create a basis for corrective actions and the planning ahead for the final 2 years of implementation. The review will be an input to a mid-term meeting we want to hold in the second part of 2011.

w w

w. R

e s

u l t

a t

s t y

r i n

g .

n o


Objective of EvaluationFormålet med evalueringen er å:• å gi LNU og bevilgende myndigheter økt kunnskap om i

hvilken grad de tiltakene som får støtte fra LNU bidrar til å nå støtteordningens prosjektmål (måloppnåelse).

• å gi LNU og bevilgende myndigheter økt kunnskap om eventuelle ikke-planlagte effekter (positive eller negative) av de tiltakene som får tilskudd.

• å gi LNU et grunnlag for å videreutvikle sin forvaltning av støtteordningen.

Formålet med evalueringen er avgrenset på bakgrunn av de økonomiske rammene for evalueringen.

w w

w. R

e s

u l t

a t

s t y

r i n

g .

n o


Organisational review:

w w

w. R

e s

u l t

a t

s t y

r i n

g .

n o

PREBEN LINDØE for BN 2006

5. Any special requirements to Team, method etc.

Team:Internal / externalSpecial expertise necessaryMethod: Level of data collectionSpecial tools to be used

NB! The action plan should not be too detailed at this point. Difficult to change.

w w

w. R

e s

u l t

a t

s t y

r i n

g .

n o

6. Outputs/ main activities

The Action plan is split into• Planning phase• Implementation• Follow up

w w

w. R

e s

u l t

a t

s t y

r i n

g .

n o

Planning phase

Outputs and activities Timeframe Responsible1. Plan of Action 03.02.12 AKV,KH;DH1. Approval Plan of Action 21.02.12 AKV1. TOR 28.02.12 AKV,KH,DH1. Select internal team members 21.02.12 SLM1. Go out for tender 29.02.12 TM/DH Implementation phase

1. Consultant chosen 15.03.12 AKV,KH1. Prepare for evaluation

1. Evaluation (by team) 16.03-30.04.12

1. Report draft 31.05.12

Completion phase

1. Quality assurance of the reports are done by relevant stakeholders

09.06.12

1. Final report 15.06.12

Example of Outputs and activities in Action Plan

Did you forget about

Learning?

7. Risk and uncertaintiesAny project meets uncertainties: e.g

w w

w. R

e s

u l t

a t

s t y

r i n

g .

n o

Risk factors Mitigating activities

An evaluation in four parts over several months lead to tiredness and lack of interest of the process among stakeholders and organization

Ensure internal communication about process and sub results along the way and continuous involvement of stakeholders.

Reputational risk in case of exposure of mistakes made by NorCross during the programme

Communication strategy is developed. Follow-up of recommendations and plan for learning is developed. NorCross initiatives to expose and learn from own mistakes decrease reputational risks (precautionary action).

Low participation by the appointed resource group and ToR groups

Involvement in the plan for the process by the members to ensure ownership. Appoint one responsible in each ToR group. Announce workshop in due time. Limited time frame and occasions for participation

Context in host countries hinders and/or complicates evaluation implementation

Research ahead and involvement of NSs

Difficulties in finding qualified consultants for evaluation implementation (technical knowledge in various areas is necessary)

Tender process through Mercell. Separate invitations (through Mercell) to tender for companies recognized by NorCross after research

9. Organisation and responsibillities

Function

Title

Tasks

Owner of evaluation

Director International Division

Responsible Anne Kirsti Vartdal Kjersti HaraldseideDagne Hordvei

Carry out activities in accordance with this PoA: -Initiate, choose consultant and team, prepare, assist evaluation team, comment on report, follow up findings.

Steering committee

Management in International division

Approve PoA and budgetApprove Report

Resource group Adviser 1Adviser 2Adviser 3Adviser 4Rep delegate section

w w

w. R

e s

u l t

a t

s t y

r i n

g .

n o

Læring – referansegruppe evalueringsavdelingen norad

Det er evalueringsavdelingen sin målsetning at evalueringsprosesser skal være deltakende og inklusive og fremme størst mulig læring. Det etableres derfor ofte en referansegruppe bestående av interessenter, berørte parter, ev. faglige ressurspersoner, fagavdeling og evalueringsavdelingen. Disse kommenterer arbeidet underveis. I den grad det er hensiktsmessig ønsker også evalueringsavdelingen å arrangere seminarer underveis og i etterkant.

w w

w. R

e s

u l t

a t

s t y

r i n

g .

n o

OECD evaluation on DAC principles• There are reason for concern regarding

the way in which lessons and experience, gained during the evaluation process, are transmitted to current operation managers.

w w

w. R

e s

u l t

a t

s t y

r i n

g .

n o

10. Budget example 1Items Activity number Budget cost (NOK)

Consultant Haiti 66%Sri Lanka 34%

240.000

Travels 55.000

Misc 5.000

Total 300.000

w w

w. R

e s

u l t

a t

s t y

r i n

g .

n o

10. Budget example 2Items Activity number Budget cost (NOK)

PREPARATION-Stakeholder meeting-M&E system updateIMPLEMENTATION-Consultant-Travels

Haiti 66%Sri Lanka 34% 240.000

55.000FOLLOW UP-Dissemination-internalisation-Misc

500050.0005.000

Total 355.000

w w

w. R

e s

u l t

a t

s t y

r i n

g .

n o

TOR – Terms of reference

w w

w. R

e s

u l t

a t

s t y

r i n

g .

n o

Action Plan versus TOR

The Action plan• Formulates the internal “project” of performing

an evaluation • It documents the approval of the decision to

perform an evaluation and the budget for it. The TOR • is a work description for the team performing

the evaluation• It constitutes only one part of the Action Planw

w w.

R e

s u

l t a

t s t

y r

i n g

. n

o

Tor Norad1 Background• Short description of the programme that will be reviewed, based on the Agreement, Programme Document(s) and

appraisal• Why the review is initiated• Team composition and leadership. (e.g. Embassy/Norad team, joint donor/Partner, external team or combination of

these) 2 Purpose, context and intended use• Description of the main purpose, context and intended use (stakeholders)3 Scope of workType of assessments may include: (for definition of the review criteria, see Annex III) • Efficiency• Effectiveness• Impact (if the programme has been operating for some years)• Relevance• Sustainability• Risk management• Particular concerns to be investigated• Audit• Anti-corruption measures4 Implementation of the review• Sources of information and methodology to be employed• Division of responsibility between the consultant/team, the Embassy, other donors and the Partner(s)• Timetable for preparation, field work and finalisation of report• Budget5 Reporting • Description of required report format • The need for an introduction summary with main conclusion on

lessons learned and recommendation(s) • Report in electronic form and/or paper, language

w w

w. R

e s

u l t

a t

s t y

r i n

g .

n o

Evalueringssprinsipper (Eval Norad web)Evalueringsprinsipper• Objektive, etterprøvbare og gjennomsiktige: Vurderinger skal

være basert på faktiske utsagn; utsagn skal være basert på troverdige (reliable) data eller observasjoner. Relevante interessenter i Norge eller mottakerlandet skal konsulteres i forbindelse med evalueringers tilrettelegging og gjennomføring, inklusive i utarbeiding av oppgavebeskrivelse og diskusjon av rapportutkast.

• Upartiske: Evalueringer skal gi en avbalansert fremstilling av styrker og svakheter. I den grad interessenter har forskjellige synspunkter skal disse tas med i evalueringen.

• Uavhengige: Medlemmer av evalueringsteamet må ikke ha vært personlig engasjert i de aktiviteter som skal evalueres. Virksomheter som gjennomfører evalueringer må ikke ha vært involvert i forberedelse eller gjennomføring av aktivitetene.

w w

w. R

e s

u l t

a t

s t y

r i n

g .

n o

Description of Project /issue to be evaluated

w w

w. R

e s

u l t

a t

s t y

r i n

g .

n o

Description of project to be evaluated

Basically a summary of project document• Especially important is the goal hierarchy and

any changes in that along the way• M&E System must be described and

commented on• Previous or other relevant evaluations should

be mentioned here• Budgets must be included too.

w w

w. R

e s

u l t

a t

s t y

r i n

g .

n o

Scope of evaluation– The boundaries of the evaluation (e.g time

period, area)– Evaluation criteria and evaluation questions

w w

w. R

e s

u l t

a t

s t y

r i n

g .

n o

Scope of evaluation

The formulation of “Evaluation questions” is one method that will end up with a precise definition of what should be evaluated. Evaluation questions represent “what one wants to know through evaluation.” •When formulation the evaluation questions it is useful to consider which of the DAC Principles for Evaluation of Development Assistance that should be covered w

w w.

R e

s u

l t a

t s t

y r

i n g

. n

o

OUTPUT 1 OUTPUT 3OUTPUT 2

One goal -LFA spread out

Activities:1.2.345

Activities:1.2.345

Activities:1.2.345

Project Goal (expected Outcome)

Development Goal (expected Impact)

OUTPUT 4

Activities:1.2.345

Mål

- og

resu

ltats

tyrin

g A

nette

P. S

imon

sen

InputsInputs Inputs Inputs

DAC Principles for Evaluation of Development Assistance (DACs evaluation Criteria)

• efficiency, • effectiveness, • impact• relevance, • sustainability.

w w

w. R

e s

u l t

a t

s t y

r i n

g .

n o

Evalueringskriterier(web Norads eval avd)

• Den enkelte evaluering bør normalt belyse følgende:• Relevans: Er aktiviteten relevant i forhold til norsk bistandspolitikks

målsetninger og strategier? Er aktiviteten relevant i forhold til mottagerlandets behov og prioriteringer? Er den relevant i forhold til det utviklingsproblem den skal se på?

• Måloppnåelse (effectiveness): Er de primære målsetninger for aktiviteten nådd? Er de planlagte resultater nådd?

• Bærekraft (sustainability): Hva er langtidsvirkningen av aktiviteten? Vil aktiviteten kunne videreføres også etter at bistandsfinansieringen er avsluttet? Er det lokalt eierskap?

• Produktivitet (efficiency): Er investeringene og driftsomkostningene berettighet? Kunne de samme resultater ha blitt oppnådd med færre midler?

• Virkning (impact): Positive og negative primære og sekundære langtidseffekter produsert av aktivitetene, direkte eller indirekte, tilsiktede eller utilsiktede. w w

w. R

e s

u l t

a t

s t y

r i n

g .

n o


Efficiency Effectiveness Impact Relevance Sustainability

Evaluation criteria:

w w

w. R

e s

u l t

a t

s t y

r i n

g .

n o


http://www.oecd.org/dac/evaluationofdevelopmentprogrammes/daccriteriaforevaluatingdevelopmentassistance.htm

EfficiencyEfficiency measures the outputs -- qualitative and quantitative -- in

relation to the inputs. It is an economic term which signifies that the aid uses the least costly resources possible in order to achieve the desired results. This generally requires comparing alternative approaches to achieving the same outputs, to see whether the most efficient process has been adopted.

When evaluating the efficiency of a programme or a project, it is useful to consider the following questions:

• Were activities cost-efficient?• Were objectives achieved on time?• Was the programme or project implemented in the most

efficient way compared to alternatives?ImpactThe positive and negative changes produced by a development

intervention, directly or indirectly, intended or unintended. This involves the main impacts and effects resulting from the activity on the local social, economic, environmental and other development indicators. The examination should be concerned with both intended and unintended results and must also include the positive and negative impact of external factors, such as changes in terms of trade and financial conditions.

When evaluating the impact of a programme or a project, it is useful to consider the following questions:

• What has happened as a result of the programme or project?• What real difference has the activity made to the beneficiaries?• How many people have been affected?SustainabilitySustainability is concerned with measuring whether the benefits of

an activity are likely to continue after donor funding has been withdrawn. Projects need to be environmentally as well as financially sustainable.

When evaluating the sustainability of a programme or a project, it is useful to consider the following questions:

• To what extent did the benefits of a programme or project continue after donor funding ceased?

• What were the major factors which influenced the achievement or non-achievement of sustainability of the programme or project?

w w

w. R

e s

u l t

a t

s t y

r i n

g .

n o



EffectivenessA measure of the extent to which an aid activity attains its

objectives.In evaluating the effectiveness of a programme or a project, it

is useful to consider the following questions:• To what extent were the objectives achieved / are likely to

be achieved?• What were the major factors influencing the achievement

or non-achievement of the objectives?

w w

w. R

e s

u l t

a t

s t y

r i n

g .

n o



ImpactThe positive and negative changes produced by a development intervention, directly or indirectly, intended or unintended. This involves the main impacts and effects resulting from the activity on the local social, economic, environmental and other development indicators. When evaluating the impact of a programme or a project, it is useful to consider the following questions:

• What has happened as a result of the programme or project?• What real difference has the activity made to the beneficiaries?• How many people have been affected?w

w w.

R e

s u

l t a

t s t

y r

i n g

. n

o



RelevanceThe extent to which the aid activity is suited to the priorities and policies of the target group, recipient and donor.In evaluating the relevance of a programme or a project, it is useful to consider the following questions:• To what extent are the objectives of the programme still

valid?• Are the activities and outputs of the programme consistent

with the overall goal and the attainment of its objectives?• Are the activities and outputs of the programme consistent

with the intended impacts and effects?

w w

w. R

e s

u l t

a t

s t y

r i n

g .

n o



SustainabilitySustainability is concerned with measuring whether the benefits of an activity are likely to continue after donor funding has been withdrawn. Projects need to be environmentally as well as financially sustainable.When evaluating the sustainability of a programme or a project, it is useful to consider the following questions:

• To what extent did the benefits of a programme or project continue after donor funding ceased?

• What were the major factors which influenced the achievement or non-achievement of sustainability of the programme or project?

w w

w. R

e s

u l t

a t

s t y

r i n

g .

n o

Evalueringsavdelingen NoradMålene for evalueringsavdelingen er å: • vurdere måloppnåelse og resultater i forhold til vedtatte planer• vurdere forutsette og uforutsette virkninger av bistandstiltak• vurdere om ressursbruken står i rimelig forhold til oppnådde

resultater• systematisere erfaringer og kunnskap for å kvalitetssikre og

forbedre fremtidig bistandstiltak gjennom gode læringsprosesser

• kartlegge kunnskapshull/ vurdere de underliggende antakelsene for ulike deler av utviklingssamarbeidet

• gi informasjon til dem som bevilger og utformer ny politikk, og allmennhetenw w

w. R

e s

u l t

a t

s t y

r i n

g .

n o

Methodology

w w

w. R

e s

u l t

a t

s t y

r i n

g .

n o

Methods for collection of info/dataQuantitativ • Quantitative Surveys• Rating

Qualitative• Desk reviews• Case studies• Qualitative surveys• Interviews

w w

w. R

e s

u l t

a t

s t y

r i n

g .

n o

Ex: MethodologyLNU ønsker at følgende metoder skal benyttes:• Intervju med utvalgte brukere av støtteordningen og deres

partnere i perioden 2004 – 2008 for å vurdere effekten av de tiltakene som har fått støtte i forhold til støtteordningens mål.

• Intervju eller spørreskjema for alle organisasjoner i Norge som har brukt ordningen i perioden 2004 – 2008 for å vurdere LNUs forvaltning av støtteordningen.

• Intervju med saksbehandlere• Intervju med medlemmer av fordelingsutvalget for

støtteordningen.• Dokumentanalyse av styringsdokumentene fortøtteordningen,

herunder retningslinjene.w w

w. R

e s

u l t

a t

s t y

r i n

g .

n o

Collecting true qualitative data

• By nature , much Qualitative data are more sensitive and personal than Quantitative data.

• While surveys can be administrated to more people in a shorter time (breadth), interviews probe for more information from fewer individuals (depth).

w w

w. R

e s

u l t

a t

s t y

r i n

g .

n o

From: Real life evaluation

Collecting true qualitative data

• Interviewees may feel uncomfortable for a variety of reasons; including cultural linguistic, age related, gender related and political reasons.

• Interviewees who are less than fully forthcoming obstruct data collection.

w w

w. R

e s

u l t

a t

s t y

r i n

g .

n o

From: Real world Evaluation

A bit of both please

Quantitative – breadth in insightquantitative surveys can be administrated to more people in a shorter time

Qualitative – depth in insightqualitative interviews probe for more information from fewer individuals (depth).w

w w.

R e

s u

l t a

t s t

y r

i n g

. n

o

NB! Please Note!

NB! Please note: • there is a continuum between

quantitative and qualitative data definition and collection.

w w

w. R

e s

u l t

a t

s t y

r i n

g .

n o

QUALITATIVE

QUANTITATIVE

Qualitative / Quantitative

Indicators / data:– Qualitiative– Quantitative

Collection methods:– Qualiatiative– Quantitative

w w

w. R

e s

u l t

a t

s t y

r i n

g .

n o

Qualitative data collection• Interviews are often characterized by:

A. Structured (all asked the same questions).B. Semi-structured (questions vary).C. Unstructured (no protocol).

• Qualitative interviewing is typically semi-structured (B) which is useful as it offers focus and flexibility but also most demanding (as one must record data while adjusting wording & order of questions, preserving focus, and follow up on conversational leads). w

w w.

R e

s u

l t a

t s t

y r

i n g

. n

o

From: Real world Evaluation

Methodology – Evaluation Norad East Africa NGO Eval. April 2011

Methodology• The evaluation team shall develop an appropriate methodology

that can respond to the information needs arising from the list of evaluation questions. The evaluation team may make use of various empirical methods, e.g. questionnaire surveys, interviews, focus group discussions and participatory rural appraisal. The presentation of the methodology should indicate any significant advantages and limitations of the chosen approach.

w w

w. R

e s

u l t

a t

s t y

r i n

g .

n o

Ex of MethodologyThe methods employed in this evaluation will include the

following: review of available documents and literature, individual interviews and focus groups discussions. It will be important to allow intended beneficiaries to tell their stories without necessarily sticking to a structured questionnaire as this will allow interviewees to release their emotions alongside the needed information. The purpose of employing several methods to gather data (triangulation) is to see whether the different sets of data material confirm or may complement each other, and hence provide a deeper understanding of the programme, its activities, effects and challenges.

w w

w. R

e s

u l t

a t

s t y

r i n

g .

n o

Sikre bruk av M&ESystemet I evalueringen

w w

w. R

e s

u l t

a t

s t y

r i n

g .

n o

Ex: Monitoring health by indicators

Outcome 2: Risk Groups adopt good practices of prevention and treatment of HIV/AIDS

# of people referred for councelling and testing

# of People reached by HIV/AIDS projects in he frame of the programme (CORE indicator)

# of test taken at local clinic pr year.

Output 1: NS engage youth in youth peer education poagramme

# of youth certified as peer educators

# of youth who have received training in peer education

Output 2: NS provide education for risk groups on sexual health (STD and HIV/AIDS)

# of people educated on sexual health

% of partcipants satisfied with the courses.

Output 3: NS have adequatly trained staff and volunteers to adress HIV/AIDS according to org. standards and in gender sensitive manner.

# of volunteers and staff participating in HIV/AIDS prevention activities in the frame of this programme (CORE Indicator)

Output 4: NS promotes HIV/AIDS voluntary testing and councelling # of information campaigns

Eksempel på M&E plan

M

ål- o

g re

sulta

tsty

ring

Ane

tte P.

Sim

onse

n

Årlig rapportm/ regnskap

(Partner)

Indikator survey

Partner

Administrativ Controller visits

(Donor)

Resultat gjennomgang /

(Joint)

2011 Baseline Jan 2012

2012 Årsrapport for 2012

i mars 2013

Februar 2013

Oktober 2012 Controller visit

August 2012 Workshop om baseline og indikatorer

2013Årsrapport for

2013i mars 2014

Februar 2014

Oktober 2013 Gjennomgang av organisasjonens administrative

systemer

2014Sluttrapport

for 2012-2014 i mars 2015

Februar 2015

Slutten av 2014 gjennomgang av

prosjekt med fokus på måloppnåelse

Team composition

w w

w. R

e s

u l t

a t

s t y

r i n

g .

n o

Team composition

Control versus learning:Internal team versus external to project

Origin/languageNorwegian, international/local base

Qualifications

w w

w. R

e s

u l t

a t

s t y

r i n

g .

n o

Internal versus External Evaluator

M

ål- o

g re

sulta

tsty

ring

Ane

tte P.

Sim

onse

n

Advantages INTERNAL Advantages EXTERNAL

Knows the situation, history, problems, processes etc

Not personally involved -easy to be objective

Understands and can interpret personal behaviour and attitudes

Free from local bias

Is known to stakeholders so pose no threat anxiety or disruption

Can bring fresh perspective and insights

Can follow up on recommendation May have broader experienceIs often less expensive and Quick to hire ( no negotiations etc)

Can serve as arbitrator or facilitator between parties

May provide mor oportunity to build local evaluation capacity

Can bring local stakeholders into contact with additional sources

ADVANTAGES

Internal versus External Evaluator

M

ål- o

g re

sulta

tsty

ring

Ane

tte P.

Sim

onse

n

Disadavantages INTERNAL Disadavantages EXTERNALHard to be objective May not know local situation

May avoid looking for facts or forming nagative conclusions tha reflect badly on community individuals

Ignorant of constraints affecting feasibility of recommendations

Tends to accept the assumptions of the stakeholders and community

May be perceived as adversary and arouse anxiety

May be too busy to participate fully May be expensive

May be part of the local community and may be constrained by conflicts of interest

Requires more time for contract negotiations, orientation and monitoring

May not be trained in evaluation methods and technical expertice

May be unfamiliar with local political and economic environment

DISADVANTAGES

The TOR

The TOR:• In planning phase the TOR is part of the tender

documents and e.g. thus describes the qualifications that is saught for the evaluation.

• After the team has been selected and contracted, the TOR is updated with the names of the team members and enclosed the Evaluation as an annex

w w

w. R

e s

u l t

a t

s t y

r i n

g .

n o

Ex of team selected team compositionTeam composition / representation:• Team leader, external consultant• YRCS programs’ coordinator• Representative of the YRCS OD department• YRCS health Coordinator• Resource persons:• The YRCS CBHD programme manager• The YRCS CBHD programme officer• President / Secretary general of Hajjah and Hodeidah branches

and sub-branches (4 people)• NRC OD advisor• SRC health advisor• DRC OD advisor• IFRC team member

w w

w. R

e s

u l t

a t

s t y

r i n

g .

n o

Reporting

w w

w. R

e s

u l t

a t

s t y

r i n

g .

n o

Reporting deliverables &formats• An inception report detailing the strategy to be utilised, defining

clear roles and responsibilities of the consultant, work plan, budget and a list of stakeholders to be consulted. This is expected within the first two days of commencement of contract;

• A draft baseline report for the governance and livelihood programme to be presented to Caritas Malawi. This is to be presented on the 8th day of the commencement of the consultancy.

• A draft report discussed between Caritas Malawi and the Consultant on the 9th day of the consultancy

• A final baseline report on governance and livelihood after incorporating comments from Caritas Malawi and Caritas Norway. This should be submitted in two hard copies and one electronic copy on the 12th day of the commencement of the consultancy.

w w

w. R

e s

u l t

a t

s t y

r i n

g .

n o

Learning

w w

w. R

e s

u l t

a t

s t y

r i n

g .

n o

Dissemination - variationsTransparent and complete dissemination. Evaluation results should be placed in public domain and widely disseminated, ensuring that information is readily accessible to all stakeholders. An initial dissemination list (Practice 5.10) should be employed to ensure the evaluation report or summaryreaches its intended audience (per the TORAppropriate dissemination. The dissemination of the evaluation report may take a variety of forms that are appropriate to the specific audience. This can include posting reports or excerpts/summaries on a community notice board or on the internet, and presentations at planning meetings, community meetings, and industry conferences. Such considerations are especially important when sharing reports with communities.

w w

w. R

e s

u l t

a t

s t y

r i n

g .

n o

Dissemination - variationsDifferentiate between Internal and external dissemination. This may be done for multiple reasons: 1) with sensitive issues that should remain internal, 2) to protect the identity and avoid or reduce any harm

to evaluation subjects, and 3) to improve the comprehensibility of the report in a

summary or simplified form for external audiences.

Disseminate through Evaluation Database. All evaluation reports should be submitted for record in international Database on evaluation

w w

w. R

e s

u l t

a t

s t y

r i n

g .

n o

Bruk av M&E info

• For at resultatmålingen skal ha noen hensikt, må resultatene drøftes, vurderes og brukes av ledelsen til å ta beslutninger om tiltak som skal forbedre virksomheten og gi bedre måloppnåelse.

w w

w. R

e s

u l t

a t

s t y

r i n

g .

n o

Mål- og resultatstyring SSØ s 8

Learning

1. Quality of learning depends on enough slow time. (opportunities)

w w

w. R

e s

u l t

a t

s t y

r i n

g .

n o

Fritt etter Knut Harald Ulland Utviklingsfondet

Learning – a crime?

‘ it is almost as if my organisation considers learning as a crime rather than a behaviour we’re trying to encourage’.A crime (or learning) needs MMO to incur:1. Motive 2. Means3. Opportunity

w w

w. R

e s

u l t

a t

s t y

r i n

g .

n o

Bruce Britton (2005)

MMO for LearningMotive:• Develop systems for acknowledging and rewarding learningMeans:• Map out where expertise lies in organisation.• Develop team working• E.g introduce methods such as mentoring, coaching, action learning and

communities of practiceOpportunity:• Create space for learning• Build time and resources for reflection and learning intor

project/programme proposals• Involve staff/partners alongside external consultants in review and

evaluation teams.

w w

w. R

e s

u l t

a t

s t y

r i n

g .

n o

B. Britton ( 2005)

Time Schedule

w w

w. R

e s

u l t

a t

s t y

r i n

g .

n o

Ex Time schedual for external mid tern review

Task/Activity Timeline 1. Procure Team By week 12: 25.03.1. Contract signed by Team By week 14: 08.04.1. Preparation work by Teamleader in Oslo (finalization of

evaluation plan, document collection, consultations with resource persons, and initial document review)

Weeks 15: 11.04 -15.04

1. Desk study/document review and fieldwork preparations

Week 17:26.04-29.04

1. Field interviews and visits

Uganda - week 21: 23.05 - 27.05 Zambia – week 22:30.05 - 03.06Mozambique – week 23:06.06- 10.06

1. Data analysis and drafting the report Week 24: 13.06-22.06.2011

1. Submission of draft to SOS Children’s Villages 23.06.20111. Draft assessed by SOS Children’s Villages Week 25:

23.06-28.061. Incorporation of comments and inputs from SOS

Children’s VillagesWeek 26:28.06-29.06

1. Submission of final report to SOS Children’s Villages Week 26:30.06.2011

w w

w. R

e s

u l t

a t

s t y

r i n

g .

n o

TOR – Terms of referenceVi finner «Den beste»

w w

w. R

e s

u l t

a t

s t y

r i n

g .

n o

- Oppfølging - har målgruppen nytte av

evalueringen?

w w

w. R

e s

u l t

a t

s t y

r i n

g .

n o

Thank you for your contribution

today!

w w

w. R

e s

u l t

a t

s t y

r i n

g .

n o

Documents

Evaluering – oppskrift på TOR