!e U.Va. Center to Promote Effective Youth Development

Using Evaluation Data to Drive Organizational Decision Making

Maryfrances Porter, PhD Associate Director

Program Evaluation and Community Consultation

Bob Andoga Director of Operations

James Pierce

Executive Director

!e U.Va. Center to Promote Effective Youth Development

Outline ∆ Decisions…decisions

∆ Data and its uses

∆ Does it mean what I think it means?

BREAK

∆ Get your group on!

!e U.Va. Center to Promote Effective Youth Development

Decisions…decisions

!e U.Va. Center to Promote Effective Youth Development

How to decide

?  How do you make every day decisions ? ?  What is hard about decision making ?

So we try to know and we try to predict…

!e U.Va. Center to Promote Effective Youth Development

Resistance

Ø  Using data for GOOD and not EVIL

Ø  Managing the unhappy and disgruntled

Ø  Getting buy-in to the process

Ø  Group decision making – perceived control

!e U.Va. Center to Promote Effective Youth Development

Data and its uses

!e U.Va. Center to Promote Effective Youth Development

One at a time, please!

ü  Only answer ONE question at a time with data.

!e U.Va. Center to Promote Effective Youth Development

How do you really know?

Organizational functioning

v  Number of clients served

v  Number of programs offered

v  Number of staff per client, etc.

!e U.Va. Center to Promote Effective Youth Development

How do you really know? Basic knowledge

v  People like what you do J

v  You are using an evidence-based practice – or – it’s worked for others

v  You are using an evidence-based practice with fidelity (you’re measuring!)

v  Clients’ behavior changes (or is good) during the intervention

v  Things were one way – you did something/something happened – things changed

v  There are community-level changes in a related public health indicator

!e U.Va. Center to Promote Effective Youth Development

How do you really know? More rigorous evaluation

v  Clients state new preferences and anticipate future behavior change

v  Clients demonstrate immediate knowledge gained (pre-post changes)

v  Change in client behavior over time

v  Client change compared to a similar client without intervention

v  Randomized intervention with comparison group (with no intervention)

!e U.Va. Center to Promote Effective Youth Development

Types of Individual Data

§  Staff reports

§  Client self-reports

§  Third party reports

§  Standardized test data, standard data collection

§  Interviews

§  Focus groups

!e U.Va. Center to Promote Effective Youth Development

Real Life

Confirmatory evidence

Convergent evidence

!e U.Va. Center to Promote Effective Youth Development

Does it mean what I think it means?

!e U.Va. Center to Promote Effective Youth Development

Thing to consider §  Did everyone change over this period of time – even if they were not in our

program (i.e., no comparison group)?

§  Are the people who participate in our program different, in important ways, from people who do not participate in our program (e.g., selection effects)?

§  Where there important external changes that might have effected programming (e.g., changes in staff, policies, a community event, etc.)?

§  Is there some specific, key factor that makes the program work other than the programming itself (e.g., a wizard effect)?

§  What actually happened to make the change (e.g., is it a fluke)?

§  Watch for unintended positive and negative consequences.

§  How much does the needle need to move to make it “real”?

§  What are alternative explanations for the finding?

!e U.Va. Center to Promote Effective Youth Development

www.ccfinfo.org Charlottesville/Albemarle Comprehensive Services Act Quarterly Outcome Report FY09, 4th Quarter

!e U.Va. Center to Promote Effective Youth Development

www.ccfinfo.org Charlottesville/Albemarle Comprehensive Services Act Quarterly Outcome Report FY09, 4th Quarter

!e U.Va. Center to Promote Effective Youth Development

http://www.fact.state.va.us/pdfs/TheFACTReport.pdf The Family and Children’s Trust Fund of Virginia Violence at Home: The FACT Report, June 2010

!e U.Va. Center to Promote Effective Youth Development

http://www.fact.state.va.us/pdfs/TheFACTReport.pdf The Family and Children’s Trust Fund of Virginia Violence at Home: The FACT Report, June 2010

!e U.Va. Center to Promote Effective Youth Development

Get your group on!

!e U.Va. Center to Promote Effective Youth Development

The Way

ü  Gather people with different perspectives

ü  Set a tone of acceptance and curiosity

ü  Review data regularly

ü  Review data over time (like good wine, it gets better with time!)

ü  Note internal and external changes that might effect data

ü  Ask someone “outside” for their thoughts

ü  Kick it around – consider different angles

ü  Generate a list

ü  Process of elimination

ü  Try something – does the outcome change?

!e U.Va. Center to Promote Effective Youth Development

The Group

v  Help the Boys and Girls Club look at data

v  Kick it around!

v  Be open!

!e U.Va. Center to Promote Effective Youth Development

What They Do Clients

v  Young people, ages 6-18

Inputs v  Caring staff with age-appropriate training in youth-development

v  Regular academic support programming

v  Fitness, nutrition, and positive decision-making activities

v  Special events and on-going clubs allow kids to engage in character and leadership development

Selected Outcomes v  Decreased number of times skipping school.

v  Decreased number of negative peers as friends.

v  Lower likelihood of starting to smoke cigarettes and drink alcohol.

v  Increased rate of on-time graduation from high school

!e U.Va. Center to Promote Effective Youth Development

The Problem

School year attendance…

!e U.Va. Center to Promote Effective Youth Development

The Data

0

10

20

30

40

50

60

70

80

90

100

6 7 8 9 10 11 12 13 14 15 16 17 18 19 20

Num

ber o

f Mem

bers

Member Age

Cherry Ave Registered Members

2010

2011

!e U.Va. Center to Promote Effective Youth Development

0%

10%

20%

30%

40%

50%

60%

< 1 yr 1 - 2 yrs > 2 yrs

Perc

enta

ge o

f Mem

bers

Years at Club

Tenure of Registered Members

2010

2011

!e U.Va. Center to Promote Effective Youth Development

0%

2%

4%

6%

8%

10%

12%

14%

16%

18%

20%

6 - 10 11 - 12 13 - 15 16 - 20

Perc

enta

ge C

omin

g 3

days

/wee

k

Age Group

Annual Visits by Registered Members

2010

2011

!e U.Va. Center to Promote Effective Youth Development

0 10 20 30 40 50 60 70 80 90 100 110 120 130 140 150 160

Career Launch Clay Tech

Digital Arts Festivals Digital Arts Suite

Game Tech YouthNet

Skill Tech: Basic Training Drama Matters Imagemakers

Imagemakers Contest Money Matters

Fine Arts Exhibit Junior Staff

Keystone Club Passport to Manhood

Torch Club Youth for Unity

Youth of the Year Jr. NBA/WNBA

Football WANNA PLAY?

SMART Girls SMART Moves

Triple Play Daily Callenges Triple Play Games

Triple Play Healthy Habits Triple Play Leadership

Number of Participants

Program Participation

2011

2010

!e U.Va. Center to Promote Effective Youth Development

0 100 200 300 400 500 600 700

Other Health

Triple Play SMART Gamesroom

Other Character

Power Hour

Project Learn

Other Academic

Number of Participants

Program Participation

2011

2010

!e U.Va. Center to Promote Effective Youth Development

What do you think?

Ø  What conclusions have you drawn?

Ø  What are the most likely next best steps?

!e U.Va. Center to Promote Effective Youth Development

Thank you

Maryfrances Porter, PhD Associate Director

Program Evaluation and Community Consultation 434/243-3698

mporter@virginia.edu

Youth-Nex http://curry.virginia.edu/research/centers/youth-nex

Program Evaluation and Community Consultation

http://curry.virginia.edu/academics/directory/maryfrances-porter

!e U.Va. Center to Promote Effective Youth Development

!e U.Va. Center to Promote Effective Youth Development