Embed Size (px)
Citation preview
Exploring how evaluation practice changes over time
Postgraduate Conference, 25th April 2015Joana Zozimo, Educational Research, Lancaster University
Why researching evaluation?
• Experience evaluation practice in four countries: Portugal; Spain; Mozambique and UK
• Short term experiences: Guinea Bissau; Cape Verde; Angola and Senegal
• PCM cycle main expertise: evaluation stage
• Evaluation resistance, misunderstanding, struggling
• Bed time story explaining researcher/evaluator
Research Problem
• Performance measurement framework places certainty and attribution (Kusek and Rist, 2004) over reflective practice, uncertainty and contribution (Mayne, 2008; Patton, 2010; Saunders et al., 2005)
• Development education (Dev Ed) sector knows very little about how evaluation is being practiced, by whom and where it is being practiced and to what effect (Henry and Mark, 2003).
• Dev Ed ultimate goals are attitudinal change; social change and/or change behaviour of its beneficiaries.
• Scholars emphasised that Dev Ed changes are more likely to be captured through a practice led-evaluation rather than a performance led-evaluation (Bourn, 2014).
What is Development education?
• active learning process, • founded on values of solidarity, equality, inclusion and co-operation. • enables people to move from basic awareness of international development
priorities and sustainable human development, • through understanding of the causes and effects of global issues • to personal involvement and informed actions. • fosters the full participation of all citizens in world-wide poverty eradication, and the
fight against exclusion. • seeks to influence more just and sustainable economic, social, environmental,
human rights based national and international policies (DARE Forum, 2004).
Dev Ed Research context : funding dependent; volatile human resources; short term activities based.
Research Problem (cont.)
My approach
• Overall research aim: explore how evaluation practice changes over time within developmen education sector.
• Research context: Youth Project evaluation practice timeline; three-year project; run by NDEC (3 groups of participants; funder-recipient context based)
• Research questions: How do development education organisations evaluate their interventions?- What do the Youth Project stakeholders do when evaluate it?- What are the internal and external influences that shape NDEC’s evaluation practice,
particularly in the Youth Project?
• What is Evaluation? What is a social practice?
What is Evaluation?
• sets or clusters of behaviours• conceptualised as social practices• forming ways of ‘thinking and doing’ • associated with undertaking evaluative activity• attributing value, complexity and appropriateness to social
interventions (Saunders et al., 2011)
• I will use the term “Evaluative Practices”
• In other words, to evaluate is to attribute value …
How Evaluation is broadly perceived
What is a social practice?
• routinized type of behaviour,• forms of bodily activities, • forms of mental activities, • ‘things’ and their use, • a background knowledge in the form of understanding, • know-how, • states of emotion and motivational knowledge (Reckwitz, 2002)
In everyday’ s language, a practice is:• a way of cooking, of working, • of investigating, of taking care of oneself or of others, • of evaluating
What is a social practice? (cont.)
Elements of Practice : Materials; Competence; MeaningDynamics of social practice: co-occurrence of practices; how they are carried by
practitioners (Shove et al., 2012)• Overlap• Interact• Compete• Collaborate• Integrate• Dominate• Or what else?..
Social Practice Theory by Reckwitz (2002) and Shove et al (2012) applied to Evaluation by Saunders (2011) in a Development Education sector.
Methodology / Research design
Social constructivist perspective; Case study research as approach , not only method! (Simons, 2009); Single in-depth
case study; Qualitative inductive approach, purposeful sampling to allow rich case to be
explored; The case: Evaluation practice of the Youth project (three-year project run by NDEC).
The case design: Units of analysis/ Embedded cases (Yin, 2003):1. Coordinators2. Staff3. Funders
Research decision: ethical decision to design three embedded cases to preserve indviduals’ anonynity and confidentiality.
Data collection overview
• Pilot study with 5 informal conversations; 3 interviews• Fieldwork in practice: 7 hr of participant observation• Observation notes; Informal conversations (n=3); documents (n=10);
Phase 1
Planning the Evaluation
Feb till May 2012
• 15 in-depth semi structured interviews,• 95 hr of participant observation spread over 3 months (weekly basis)• 24 hr of non participant observation spread over 2 months (event basis)• Interviews notes (n=15); observation notes;
Phase 2
Evaluation off the ground
Sept-Nov 2012
• 3 in-depth interviews; 3 informal conversations• 4 documents analysed• Interview notes (n=3); Informal conversations (n=3)
Phase 3
Evaluation reporting
March-April 2013
My Data set/ Approach to analysis
6 Informal conversations notes/research diary 18 interviews transcripts (fully transcribed) 14 documents 126 hr observational field notes /research diary
Case “story” account (Simons, 2009); Transforming data (Wolcott, 1994)“Thick” description (Geertz, 1973); Longitudinal analysis (Saldana, 2002)
Atlas TI software; coding and categorisation; seek patterns and connections; Three cycles of coding - Content analysis, value coding analysis, longitudinal analysis; Social practice theoretical lenses applied to focus on what different groups do on a
daily basis – coordinators; staff and funders;
Informed by literature on: Moments of evaluation (McCluskey et al, 2008) Provisional stabilities (Saunders, 2005)Dynamics of practice (Shove et al., 2012)
The Case Evaluation Practice Timeline (WIP)
Example imageYouth
Project approved
DfID’s official
letter and guidelibes
Stage 1Planning
Evaluation
GMA’s evaluation mandate
Practice change episodes (PCE)DfID’s
evaluation
reviewed
My presence
in the fieldwork
Stage 2
Getting
Evaluation “off the
ground”
Stage 3 Reporti
ng Evaluati
on
Dev Ed
Political
context
change
PCE - Funders contrasting evaluation approaches
2009/20102011 3/12 9/12 1/13
The Case Evaluation Practice Timeline (WIP)
Example image
FindingDesigning
the Project
Stage 1Planning
Evaluation
Practice change
episodes
Researcher presence in
the fieldwork
Stage 2Getting
Evaluation “off the ground”
Stage 3 Reporting Evaluation
2009/2010 2011 3/12 9/12 1/13
Main Themes
• Absence of evaluation culture (dishonesty, fear, scepticism)• Retrospective reflection
Finding
Designing the Project
• Evaluation language/discourse• Forced/Imposed practice
Stage 1 Planning
Evaluation
• Resistance • Power (funders’ relationship)• Managerialism• Co-occurrence of practices
Stage 2
Evaluation “off the ground”
• Expectations• Fragmented evaluation practice• Competence
Stage 3
Reporting Evaluation
My account/ research story
• The Grant Management Agency only administered the grant, they weren’t development education specialists. It’s like asking your bank what are the best sausages, you know...they don’t know [laughs]. They might eat it, but they don’t know; you know what I mean...they are not education people; they are consultants who are administering the grants. (Mary, project worker, 7:77, emphasis added)
My account/ research story
Reflection enables practitioners to notice how evaluation practice has changed over time; and to learn from it !
My account/ research story
When a new actor [GMA] appear within the funders/recipient context, evaluation practice changes (the way a practice is carried by the practitioners is altered, because new actors
have occupied that space in the field).
My account /research story
A fragmented practice changes the way evaluation is practiced over time.
My account /research story
Research participants are expected to have competence in evaluation, to respond to funders mandatory requirements
My contribution is to the evaluationpractice debate Theoretical contribution – extending evaluation practice theory by using an
advance theoretical framework of SPT applied to a new domain of Dev Ed.
Implications for Practitioners- add fresh insights on evaluative practices within Dev Ed domain; dynamics of social practice places reflective practice at the center of evaluation practice.
Research Impact – gain momentum of 2015 Year of Evaluation http://mymande.org/evalyear/Declaring_2015_as_the_International_Year_of_Evaluation
My reflective space / Research journey Writing as a sense making tool – despite the use of Atlas, it was only when
I start writing on a daily basis that I deeply engaged with my materials.
Writing as interpretation (Richardson, 1994).
Writing retreats (Rowena Murray) – learning the dynamics of academic writing; create habits; practice writing in a peer group; supportive and contained environment; create your own writing group.
More resources: • Rowena Murray work http://www.rowenamurray.org/
• FASS writing retreat http://www.lancaster.ac.uk/fass/gradschool/training/
• Thesis whisperer http://thesiswhisperer.com/
References
Bourn, D. (2014) The Theory and Practice of Global Learning. Development Education Research Centre.
DARE Forum. (2004). What is DARE. DEEEP - Developing Europeans’ Engagement for the Eradication of Global Poverty . Retrieved from http://www.deeep.org/what-is-dare-.html
Geertz, C. (1973) The Interpretation of cultures. [New York (N.Y.)]: Basic Books.
Henry, J., Mark, M.M. (2003) Toward an agenda for research on evaluation. In C. A. Christie, ed. The practice-theory relationship in evaluation. Jossey-Bass.
Kusek, J.Z., Rist, R.C. (2004) Ten Steps to a Results-Based Monitoring and Evaluation System: A Handbook for Development Practitioners. World Bank Publications.
Mayne, J. (2008) Contribution analysis: An approach to exploring cause and effect.
McCluskey, A. (2011). Evaluation as deep learning: a holistic perspective on evaluation in the Pallete project. In Reconceptualising evaluative practices in HE (Saunders et al.). Open University Press.
Reckwitz, A. (2002). Toward a Theory of Social Practices: A Development in Culturalist Theorizing. European Journal of Social Theory, 5(2), 243–263
Patton, M.Q. (2010) Developmental Evaluation: Applying Complexity Concepts to Enhance Innovation and Use . Guilford Press.
Saldana, J. (2002) Analyzing Change in Longitudinal Qualitative Data. Youth Theatre Journal 16(1). 1-17
References
Saunders, M., Charlier, B., Bonamy, J. (2005) Using Evaluation to Create ‘Provisional Stabilities’. Evaluation. 11(1), 37–54.
Saunders, M., Trowler, P., & Bamber, V. (2011). Reconceptualising Evaluative Practices in HE (1st ed.). Open University Press.
Simons, H. (2009) Case study research in practice. Los Angeles ; London: SAGE.
Shove, E. (2012) The dynamics of social practice: everyday life and how it changes. Los Angeles: SAGE.
Wolcott, H.F. (1994) Transforming qualitative data : description, analysis, and interpretation. Thousand Oaks, Calif.: Sage Publications.
Yin, R.K. (2003) Case study research: design and methods. 3rd ed. Thousand Oaks, Calif: Sage Publications.
