3/23/10

1

Empowering Youth To Be Evaluators: Involving young people in evaluating

informal education programs

Amy Grack Nelson, Evaluation & Research Associate Science Museum of Minnesota

Overview

  Overview of participatory evaluation

  Participatory evaluation examples

  Sampling of interactive techniques

What is participatory evaluation?

3/23/10

2

It’s All About Utility

  Utility - one of the four essential features of all evaluations

(Joint Committee on Standards for Educational Evaluation, 1994)

  A way to help ensure use is to increase the primary intended users’ level of participation in the evaluation. (Cousins & Earl, 1995; Patton, 2008)

Participatory Evaluation

“Applied social research that involves trained evaluation personnel and practice-based decision makers working in partnership.” (Cousins & Earl, 1995, pg. 8)

Core purpose increasing use

Characteristics of Participatory Evaluation  Balanced control of evaluation process   Involvement of primary users  Extensive participation throughout the evaluation

(Cousins & Earl, 1995; Cousins & Whitmore, 1998)

Interactive Evaluation Practice Continuum

(King & Stevahn, 2002)

3/23/10

3

Benefits of Participatory Evaluation

Increases use of evaluation results by:  Enhancing relevance of the evaluation   Increasing understanding of the data   Increasing ownership of the findings

(Cousins & Whitmore, 1998; King & Stevahn, 2002; Patton, 2008)

Evaluation capacity building  Develop analytic and evaluative skills  Stakeholders develop a more “critical eye”

(Cousins & Earl, 1992, 1995)

Science Museum of Minnesota’s Kitty Andersen Youth Science Center

Participatory Evaluation Examples

Kitty Andersen Youth Science Center’s Park Crew

Facilitate earth science and environmental education activities in the Big Back Yard and on outreaches

3/23/10

4

Example 1: Summative Evaluation

Youth will…  Learn about water related earth surface processes  Develop teaching skills  Learn about related science, technology, engineering, and

math (STEM) careers

Evaluation design  Observations and interviews of youth staff at the

beginning and end of summer  Evaluation workshop to engage youth in results

Reviewing the Work

Keep/Change Discussion

3/23/10

5

Incorporating Evaluation Data

How often youth talked about why something is considered a pollutant (n=27)

Keep/Change Discussion

  Engages users with data to think about successes and areas of improvement

  Can be used to generate recommendations

  Takes considerable amount of time

Benefits Limitations

3/23/10

6

Youth Benefits

Meaningful involvement in evaluation can help youth develop higher order thinking skills, specifically analytic and evaluative skills. (London et al., 2003)

  Youth became more reflective of their work.

  Youth comments reflected increased knowledge of the activities and confidence in sharing that knowledge with visitors.

  Youth had a stronger sense of ownership and control.

Adult Staff Benefits

  Provided important feedback about the crew’s work.

  Gained deeper understanding of the participants’ experience and could proactively identify and respond to their needs.

  Increased understanding of evaluation and ability to interpret data and generate recommendations.

  Developed capacity to include participatory evaluation in future work.

3/23/10

7

Outcomes of the Process

Participants experience a sense of empowerment and pride when they have an influence on the way programs are run and see their ideas acted upon.

(Checkoway et al., 2003; Horsch et al., 2002; London et al., 2003)

  Youth used suggestions to develop their own training.

  They created a visitor survey and collected data.

  Youth shared their ideas with a museum operations staff member.

Example 2: Formative Evaluation of Outreaches

Evaluation Process

Identify daily objectives

Craft survey questions Pilot surveys

Discuss pilot data and revise

surveys

Administer surveys

Enter and code data

Analyze and discuss data

Generate recommendations

Improve outreach activities

3/23/10

8

Organizational Requirements for PE

  Evaluation must be valued

  Sufficient time and resources

  Commitment to organizational learning as a means to improvement

  Motivated individuals

  Interest and ability to learn evaluative skills

(Cousins & Earl, 1992)

Evaluator Requirements for PE

  Sufficient technical and facilitator skills

  Accessible for participatory activities and support

  Necessary resources and time

  Serve an instructional role

  Motivation and commitment to participate

  A tolerance for imperfection

  Flexibility (Burke, 1998; Cousins & Earl, 1992; King, 1998)

3/23/10

9

A sampling of interactive methods to gather and discuss evaluation data

Interactive Techniques

What youth do when they get stuck on a project (n=8)

3/23/10

10

Interactive Graphs

  Quick data collection   Everyone can see the

process and results   Can be used as a

starting point for deeper conversations

  Can see data by various characteristics

  People may be influenced by others

  People may be hesitant to place a rating where no one else has

Benefits Limitations

(King, 2009)

Carousel Sheets

  Alternative to traditional brainstorming

  Lots of information in a short timeframe

  Quick way to see patterns   Promotes high involvement   Involves users in analysis

  Participants may influence each others’ responses

  Tend to get first responses and gut reaction; not deep and thoughtful

  Responses may be too brief

  May need to reanalyze some of the data

Benefits Limitations

(King, 2009)

3/23/10

11

Amy Grack Nelson: agnelson@smm.org

To see a sampling of evaluation reports visit: www.smm.org/researchandeval

Questions?