The Importance of outcomes and evaluation in service delivery

Yfoundations 10th July 2012

Prof Ilan Katz (SPRC)

Will cover

Why outcomes are important Evaluating for outcomes Monitoring and evaluation Types of evaluation Some key methods Conclusions

Why are outcomes important

Most services focus on outputs: Numbers of clients seen Numbers of service events Unit costs

However this does not measure impacts ie whether the service makes a difference can create perverse incentives.

Outcome measurement can improve service delivery.

How does evaluation improve services

Important to know what impact the service is actually making on clients and why.

Better understanding of who benefits most from the service and which aspect of the service is most beneficial.

Results are fed back into the service so that lessons can be learned about elements of effective practice.

Outcomes and accountability

Accountability increasingly focused on outcomes rather than outputs.

Funding decisions now rely on cost effectiveness as well as assessment of need.

Outcomes and evaluation can also increase accountability to service users, as participants and recipients.

Evaluation and monitoring

Monitoring – regular collection of data which is used for quality assurance, adherence to standards and compliance.

Evaluation – one off or episodic activity focusing on effectiveness of implementation, processes, outcomes and/or cost effectiveness.

Both are important and can build on each other.

Theory of change or program logic

Links inputs processes, outputs and outcomes.

Explains why activities should lead to particular outcomes.

Needs to be backed up by theory and empirical evidence and be specific.

Explains the link between inputs, processes and outcomes.

Process evaluation

Is service doing the things it is supposed to do in the most efficient way possible?

Has the program been implemented as planned?

Is it targeting the right people? What are the barriers to accessing the

service? How well is it working with other services? What are clients’ and other stakeholders’

views of the service?

Intermediate Outcome:Young people at risk of homelessness are provided with safe accommodation and given the

opportunity to address issues leading to homelessness

Ultimate Outcome:Young people find stable accommodation and are able to participate actively in society to their maximum potentioal

Situation· Limited access to

Community specialist services

·High demand for service

·Young people unable to access mainstream housing services

·Young people leaving families and becoming vulnerable to exploitation and abuse

Inputs$X put into youth homelessness · Staffing resources

Policies – Commonwealth and NSW

Other services

Client Group: Young people at risk of homelessness

Inputs$X put into youth homelessness · Staffing resources

Policies – Commonwealth and NSW

Other services

Client Group: Young people at risk of homelessness

OutputsWhat services will provide:· counselling·initial assessments·Temporary accommodation·brief interventions·Benefits advice·Peer support·providing education and support to referrers

Who the services will interact with:·Young people with acute housing issues ·families and carers·other members of the broader youth services·primary referrers and other service partners

OutputsWhat services will provide:· counselling·initial assessments·Temporary accommodation·brief interventions·Benefits advice·Peer support·providing education and support to referrers

Who the services will interact with:·Young people with acute housing issues ·families and carers·other members of the broader youth services·primary referrers and other service partners

ImpactsFor children and adolescents:· Improvement in housing situation ·Safety·Improvements in educational/vocational participation·Improvements in social functioning·Improvements in financial stability·Reduction in wait time for service·Better interaction with family ·Improvement in health and mental health

For homeless Service System:·Lower numbers of homeless youth·Better inter-agency collaboration·Longer term engagement with vulnerable clients

For Referrers:·Clear referral pathway·Improvement in collaborative care

ImpactsFor children and adolescents:· Improvement in housing situation ·Safety·Improvements in educational/vocational participation·Improvements in social functioning·Improvements in financial stability·Reduction in wait time for service·Better interaction with family ·Improvement in health and mental health

For homeless Service System:·Lower numbers of homeless youth·Better inter-agency collaboration·Longer term engagement with vulnerable clients

For Referrers:·Clear referral pathway·Improvement in collaborative care

9

Methodologies

Administrative data (employment, health, housing)

Worker assessments, case files Use of standard validated surveys

Self completed surveys pre and post questionnaires Satisfaction, wellbeing, circumstances

Qualitative methodologies/action research

Outcomes

Immediate Satisfaction, appropriate referral

Intermediate Improved wellbeing, skills, housing

Long term Home ownership/stability, employment, social

networks

Challenges of measuring outcomes

When? Should you follow up clients?

How? Questionnaire, feedback, admin data

What? How to define a good outcome

Comparison and benchmarking Attribution

How do you know it was your service making the difference?

Who should do the evaluation?

In house Advantages

• Cheaper and more tailored to agency needs.

Disadvantages• Lack of credibility and independence.• Need organisational expertise

External Advantages

• Independent, authoritative – provides feedback

Disadvantages• Expensive and potentially burdensome, potential

embarrasment

Ilan KatzSocial Policy Research Centre

Matthew GrayCAEPR, ANU

Ilan.katz@unsw.edu.au

www.sprc.unsw.edu.au www.anu.edu.au/caepr

G2 Western CampusUniversity of New South Wales Kensington 2052NSW, Australia+61 2 9385 7810