Embed Size (px)
Citation preview
Measuring capacity building: mission impossible?
Tine Verdonck, Bénédicte Fonteneau (HIVA) and Bart Criel
1
The subject of our discussions…
• Academic trias– Research
– Teaching
– Service Delivery
• Service delivery: multiform– Consultancy work overseas
– Variety of services delivered to (Belgian) general public, civil society organisations, public health authorities (cfr. “steunpuntfunctie” DGD)
– Capacity building via Institutional Collaborations (ICs)
– …
2
Purpose of session
• Share views on place & importance of Capacity Building (CB) as academic activity
• Reflect on ways/methods how to measure itseffect(s).
Hypothesis: complexity of CB not properly capturedby instruments currently used
3
Programme
• Intro (Bart): 10’
• Some data on our ICs (Tine): 10’
– Quick reactions: 5’
• Outcome Mapping (Bénédicte): 20’
– Quick reactions: 5’
• Most Significant Change (Tine): 5’
• Discussion on sample of MSC stories collected (Bart and Tine): 30’
• Final thoughts (Bart): 5’
4
Some historyITM from 1990 till today
• 1990: a small ENOV / URESP– Training (ICHD): main project
• “Critical mass”
– Research: mainly action-research geared to change, few publications
– Service delivery (incl. capacity building) central in culture of unit:
• “Societal relevance”
• Relevance for decision-makers
• 2012: a big Public Health Department– Training (MPH HSMP-MDC):
still important, but less central
– Research: explosion in publications, action-research marginalized
– Service delivery (incl. capacity building)
• Has “objectively” strongly increased
• But less prominent in departmental culture and policies than in past
5
A “contradiction” in current ITM policy & discourse?
• De facto, we have never done as much CB as today
• But ITM discourse…
– vàv outside world: “ITM is different from other academic institutions”
– Internally: service delivery/capacity buidling is not a priority, not a core task, to be phased out, “we are not an NGO”
6
Evaluation / recognition academic work at ITM – as of today
• Research (publications) +++, Teaching +/++, SD/CB +/-
• Our (departmental) “weaknesses”
– Is there consensus within department on importance of SD/CB as academic activity à part entière ?
– Voluntaristic discourse, but no instrumentarium / set of indicators to measure effect of SD/CB
7
Capacity Building
ITM 2020
Working Group CB
8
Something to strive for
Interwoven with research & training
Long-term process
What is capacity building about?
9
Dimensions of capacity building
Scientific
capacityManagerial
capacity
Institutional
capacity
Capacity
Building
Adapted from Patrick Develtere, HIVA/KUL10
FUTURE
TODAY
Capacity building: now and in the future
North
South
From rich to poorFrom knowledgeable to less knowledgeableA finite process
South
NorthNorth
South Alliances between equal partners
Mutually beneficial
Long lasting
11
Powerful justification of ITM involvement in capacity building
Ethical reasons: solidarity
Pragmatic / opportunistic reasons: ITM needs strong partners & “fields” to do research
Technical reasons: ITM’s expertise in capacity building, health and health care (“excellence”?)
ITM and capacity building after 2020? Probably (still) the case, here and there, but with highly variable outlook depending on context
Our message
12
A framework to analyse Capacity Building
13
Capacity BuildingPotter C & Brough R (2004)
Systemic Capacity Building: a hierarchy of needs Health Policy & Planning; 19(5): 336-345
Definition?
– “There is a need for CB” is an as unuseful diagnosis as “the patient is feeling unwell”
– Consultant-speak
– Over-pompous synonym for training
– …
-> Lack of clarity
14
Capacity building (follow.)Potter & Brough (2004)
“Enable programme execution independent of changes of personalities, technologies, social structures and resource crises”
“Develop sustainable and robust systems”
15
Systems Capacity: a hierarchy of needs
Potter & Brough (2004)
Structures, Systems and
Roles
Staff & Infrastructure
Skills
Tools
Capacity Pyramid
enable
effective
use of…require …
16
Systems Capacity: a hierarchy of needs
Potter & Brough (2004)
Structures, Systems and Roles
Staff & Infrastructure
Skills
Tools
“Easier” and
more
technical
“Harder” and
more socio-
cultural
Time to implement change
17
Indicators of the Framework Agreement
Project cycle management - indicators
• How do we verify success?
• Describe the objectives in operationally measurable terms (quantity, quality, time)
• Basis of monitoring and evaluation system
Our indicators
• Framework Agreement
• Subprogramme: Institutional Collaboration
• Period: 2008-2010
• Total 381; objectives 57 and results 314
• Per project; range 10-61 and mean 25
Domain of FA indicators
Management/Supporting services
Services/GRIPP/Networking
Training
Research
PI from Department Public Health PI from other Department
+ output in numbers
People• Master and PhD scholarships awarded and degrees obtained • Short course scholarships awarded and certificates issued• Exchanges and missions• ...Publications• Publications in international peer-reviewed journals; other publications• Theses (master and PhD)• ...Products• CD-ROM, digital atlas, web-based course, online-modules, website• Patent: application or obtained; test/method developed• ...
Publications Framework Agreement
• Sources of references: reports of FA 1, 2, and 3
• Including only PubMed publications
• Removing duplicates
729 publications
Number of publications per year
Nr of publ. per year according to the affiliation address of the first author
Tag cloud of key words
Questions raised
• Relevance in partner countries?
• Who determines research agenda? Who does what?
• What happens if collaboration stops?
• Are publications a good indicator of capacitybuilding?
Level of FA indicators in hierarchy
Theoretical model Framework agreement (IC; FA3-I)
Structures, Systems and Roles
Staff & Infrastructure
Skills
Tools
“Easier” and more technical
“Harder” and more socio-
cultural
Time to implement change
Formulation of FA indicators: examples of “activity done”
• Quantité et qualité des ressources documentaires acquises par le centre de documentation
• Routine testing for second line drugs standardised (2012)
• Gene mutations related to resistance investigated (2013)
• Work-based health systems fellowship programme fully established
• One dissemination workshop/year for local MOH staff
Formulation of FA indicators: examples of “changing influence”
• Publications describing the positive impact of measures recommended by XXX to other national institutions
• Policy guidelines take into account results of studies conducted at XXX
• XXX attracts a cadre of nationally renowned faculty
• Level of satisfaction of the health service managers with alumni performance and initiatives has increased
• Le nombre de cadres gestionnaires formés à XXX et exerçant des fonctions de gestion de haut niveau dans le système de santé
Formulation of FA indicators
Changing influence
Activity done
Most significant change
An experiment
Steps
1. Collecting stories (done)
2. Selecting most significant of stories (now)
3. Feeding back results of selection (later)
Most Significant Change Technique
• What? Making sense of complex information; participatory; qualitative
• When to use? Complex participatory programmes; emergent outcomes; social change
• When not to use? Accountability reports, public relations, to capture expected change
• Characteristics? Identifies values; bridges cultures; analysis > data collection; rich picture
Best illustration of what we want to achieve?
1. PhD student learned to analyse data2. Invitations to participate in consultancy & reflection3. Launch of new training programme4. PhD student involved in grant writing5. Positive feedback on newsletter6. PhD student submitted PhD proposal7. Organization of new training programme8. Inauguration alumni association9. We got access to data on births and deaths10. PhD student – discussion about draft manuscript
MSC: group discussion
• Reading out of 10 “stories” recently collected
• Discussion
– Which story is a good / the best illustration of what we wish to achieve with CB?
– Why?
– Is this currently captured by the current evaluationinstrumentarium that is used?
37
38
Final thoughts
39
Systems Capacity: a hierarchy of needs
Structures, Systems and Roles
Staff & Infrastructure
Skills
Tools
“Easier” and
more
technical
“Harder” and
more socio-
cultural
Time to implement change
40
PCM/
Logframes
Other
methods
(OM, MSC)
quant
qualit
Way forward?
• Evaluation questions: measure not only what we do (=production), but also what we induce (=change in processes)
• More room for “soft” things: values, trust, affect, self-confidence,…
• As a department, familiarise with other methods that complement PCM/Logframe
• Introduce this enriched evaluation framework in forthcoming FA4
41
42
Statements put forward to Director
• 5 statements on Service Delivery / Capacity Building
• Scaling 1 (strongly disagree) to 5 (strongly agree) by “A” & “B”
43
A: 3B: 3
• Stelling 1. Dienstverlening aan ontwikkelingslanden -waaronder onze activiteiten mbt institutionele samenwerking en steunpuntfunctie - zit ingebakken in het DNA van het ITG en haar personeel. Het was/is bepalend in de keuze van velen om op het ITG te komen werken
44
A: 1B: 1
• Stelling 2. Gegeven stelling 1, zou het van slecht personeelsbeleid getuigen om deze activiteiten niet als volwaardig te erkennen en te valoriseren.
45
A: 4B: 4
• Stelling 3. De gangbare methodieken om dienstverlening/institutionele samenwerking te meten / evalueren slagen er onvoldoende in om de complexiteit en de (eventuele) impact van deze activiteit te capteren.
46
A: 4B: 5
• Stelling 4. Het ITG moet als ambitie hebben om naast een "Centre of Excellence" in onderzoek (en onderwijs), ook een “Centre of Excellence” te zijn in het domein van de dienstverlening/ institutionele samenwerking.
47
A: 2B: 1
• Stelling 5. De huidige druk in onze samenleving op academische instellingen om steeds meer en sneller meetbare outputs (“productie”) te bereiken in hun onderzoek, gaat ten koste van de maatschappelijke dienstverlenende rol van deze instellingen.
48
