The Importance of Measuring Outcomes

  • View

  • Download

Embed Size (px)


Professor Ilan Katz


  • 1. The Importance of outcomes and evaluation in service delivery Yfoundations 10th July 2012 Prof Ilan Katz (SPRC)

2. Will cover Why outcomes are important Evaluating for outcomes Monitoring and evaluation Types of evaluation Some key methods Conclusions 3. Why are outcomes important Most services focus on outputs: Numbers of clients seen Numbers of service events Unit costs However this does not measure impacts iewhether the service makes a difference can create perverse incentives. Outcome measurement can improve servicedelivery. 4. How does evaluation improveservices Important to know what impact the service isactually making on clients and why. Better understanding of who benefits mostfrom the service and which aspect of theservice is most beneficial. Results are fed back into the service so thatlessons can be learned about elements ofeffective practice. 5. Outcomes and accountability Accountability increasingly focused onoutcomes rather than outputs. Funding decisions now rely on costeffectiveness as well as assessment of need. Outcomes and evaluation can also increaseaccountability to service users, as participantsand recipients. 6. Evaluation and monitoring Monitoring regular collection of data whichis used for quality assurance, adherence tostandards and compliance. Evaluation one off or episodic activityfocusing on effectiveness of implementation,processes, outcomes and/or costeffectiveness. Both are important and can build on eachother. 7. Theory of change or program logic Links inputs processes, outputs andoutcomes. Explains why activities should lead toparticular outcomes. Needs to be backed up by theory andempirical evidence and be specific. Explains the link between inputs, processesand outcomes. 8. Process evaluation Is service doing the things it is supposed todo in the most efficient way possible? Has the program been implemented asplanned? Is it targeting the right people? What are the barriers to accessing theservice? How well is it working with other services? What are clients and other stakeholdersviews of the service? 9. Ultimate Outcome:Young people find stable accommodation and are able to participate actively in society to their maximum potentioalIntermediate Outcome:Young people at risk of homelessness are provided with safe accommodation and given theopportunity to address issues leading to homelessnessImpactsFor children and adolescents: Improvement in housing situationOutputsSafetySituation What services will provide:Improvements in educational/vocationalcounselling participation Limited access toinitial assessmentsImprovements in social functioningCommunity specialist Inputs Temporary accommodationImprovements in financial stabilityservicesbrief interventions $X put into youth Reduction in wait time for serviceBenefits advice High demand for service homelessnessBetter interaction with familyPeer supportStaffing resources Improvement in health and mental health providing education and support to Young people unable to Policies referrers For homeless Service System:access mainstreamhousing services Commonwealth andLower numbers of homeless youth NSWWho the services will interact with: Young people with acute housing Better inter-agency collaboration Young people leaving issues Longer term engagement with vulnerable Other servicesfamilies and becomingfamilies and carersclientsvulnerable to exploitation other members of the broader youth Client Group: Youngand abuse servicesFor Referrers: people at risk of homelessnessprimary referrers and other service Clear referral pathwaypartners Improvement in collaborative care 9 10. Methodologies Administrative data (employment, health,housing) Worker assessments, case files Use of standard validated surveys Self completed surveys pre and post questionnaires Satisfaction, wellbeing, circumstances Qualitative methodologies/action research 11. Outcomes Immediate Satisfaction, appropriate referral Intermediate Improved wellbeing, skills, housing Long term Home ownership/stability, employment, socialnetworks 12. Challenges of measuringoutcomes When? Should you follow up clients? How? Questionnaire, feedback, admin data What? How to define a good outcome Comparison and benchmarking Attribution How do you know it was your service making thedifference? 13. Who should do the evaluation? In house Advantages Cheaper and more tailored to agency needs. Disadvantages Lack of credibility and independence. Need organisational expertise External Advantages Independent, authoritative provides feedback Disadvantages Expensive and potentially burdensome, potential embarrasment 14. Ilan KatzSocial Policy Research CentreMatthew GrayCAEPR, Western CampusUniversity of New South WalesKensington 2052NSW, Australia+61 2 9385 7810