Formative Evaluation of the Performance Indicators ... · to the Ministry and the field ... entry, data importing and report generation training. Ontario Tobacco Research Unit

Formative Evaluation of theFormative Evaluation of thePerformance Indicators MonitoringPerformance Indicators MonitoringSystem (PIMS) for the Smoke-freeSystem (PIMS) for the Smoke-free

Ontario StrategyOntario Strategy

Gala Arh, Robert SchwartzGala Arh, Robert Schwartzand John Garciaand John GarciaOntario Tobacco Research UnitOntario Tobacco Research Unit

Ontario Tobacco Research Unit

The purpose of performance measurementThe purpose of performance measurementis not only to develop performance indicatorsis not only to develop performance indicatorsbut to establish a system to measure resultsbut to establish a system to measure resultswhile improving management (Yang & Jun,while improving management (Yang & Jun,2007)2007)


Learning ObjectivesLearning Objectives

Recognize the context for development ofRecognize the context for development ofOntarioOntario’’s web-based Performance Indicators web-based Performance IndicatorMonitoring System (PIMS)Monitoring System (PIMS)

Understand the process for developing PIMSUnderstand the process for developing PIMSand be aware of the underlying systemand be aware of the underlying systemstructurestructure

Identify early implementation issues,Identify early implementation issues,important to consider when developingimportant to consider when developingsimilar systemssimilar systems

Learn from evaluation resultsLearn from evaluation results


Performance measurement has beenPerformance measurement has beenpromoted as a utilitarian instrument for localpromoted as a utilitarian instrument for localgovernment management since the 1940sgovernment management since the 1940s((StreibStreib & & PoisterPoister, 1999), 1999)

To assist with government management,To assist with government management,performance measurement systems mustperformance measurement systems mustflow strategic information to Ministries toflow strategic information to Ministries toassist in decision making processesassist in decision making processes


Why Develop a Performance MeasurementWhy Develop a Performance MeasurementSystem for Ontario?System for Ontario?

In 2004-05, Ontario implemented the comprehensiveIn 2004-05, Ontario implemented the comprehensiveSmoke-free Ontario StrategySmoke-free Ontario Strategy

$60M annual investment for 2007/08; six fold increase in$60M annual investment for 2007/08; six fold increase inMinistry of Health Promotion funding since 2003/04.Ministry of Health Promotion funding since 2003/04.

Program management responsibilities are greatlyProgram management responsibilities are greatlyexpanded:expanded: Decentralized program/service delivery modelDecentralized program/service delivery model 196 programs funded through local and provincial196 programs funded through local and provincial

transfer payment agencies to deliver cessation,transfer payment agencies to deliver cessation,prevention, and protection programmingprevention, and protection programming


Ministry Requirements for a PerformanceMinistry Requirements for a PerformanceMeasurement SystemMeasurement System

Be work and time savingBe work and time saving Collect program management data usefulCollect program management data useful

to the Ministry and the fieldto the Ministry and the field Provide ready access to current programProvide ready access to current program

data, including aggregate datadata, including aggregate data Provide centralized file management andProvide centralized file management and

online access to online access to workplansworkplans and reports and reports Be readily adaptable and expandableBe readily adaptable and expandable


Designing PIMS to Meet Diverse NeedsDesigning PIMS to Meet Diverse Needs

ProcessProcess RulesRules A glance at what emergedA glance at what emerged Formative evaluation findingsFormative evaluation findings


PIMS Development and ImplementationPIMS Development and ImplementationProcessProcess

1.1. Collaboratively, developed programCollaboratively, developed programindicators.indicators.

2.2. Ministry of Health Promotion approvalMinistry of Health Promotion approvalobtained for all indicators.obtained for all indicators.

3.3. All users participate in work planning, dataAll users participate in work planning, dataentry, data importing and report generationentry, data importing and report generationtraining.training.


PIMS RulesPIMS Rules

User-based and user-friendlyUser-based and user-friendly DynamicDynamic ResponsiveResponsive Work-savingWork-saving Easy to readEasy to read Beyond Indicators: Ample room for proseBeyond Indicators: Ample room for prose


PIMS specificitiesPIMS specificities

280 280 useridsuserids have been defined in PIMS with have been defined in PIMS withmore than 130 distinct user rolesmore than 130 distinct user roles

6 distinct Public Health Unit (PHU) and 176 distinct Public Health Unit (PHU) and 17Transfer Payment Agency (TPA) programsTransfer Payment Agency (TPA) programs

PIMS functionality includes: work planning,PIMS functionality includes: work planning,performance monitoring, interim and finalperformance monitoring, interim and finalreporting, and variable time reportingreporting, and variable time reporting

A total of 1271 indicators in PIMSA total of 1271 indicators in PIMS


Typical content flow between agencyTypical content flow between agencyand MHPand MHP


Data Entry and Importing in PIMSData Entry and Importing in PIMS

Easy to use templatesEasy to use templates Imports data from existing electronicImports data from existing electronic

databases thus circumventing double datadatabases thus circumventing double dataentryentry

Frequency of data entry is program specificFrequency of data entry is program specific


Performance Reporting FunctionalityPerformance Reporting Functionality

Separate reports for each indicator for anySeparate reports for each indicator for anytime periodtime period

Secondary time period reportingSecondary time period reporting Comparison to Tobacco Control AreaComparison to Tobacco Control Area

Network (TCAN) averageNetwork (TCAN) average Comparison to provincial averageComparison to provincial average Bar and line graphs for each indicatorBar and line graphs for each indicator


Semi-annual and Annual ReportsSemi-annual and Annual Reports

All Ministry required reporting is generatedAll Ministry required reporting is generatedthrough PIMSthrough PIMS

Annual and semi-annual reports areAnnual and semi-annual reports areaggregations of monthly reports withaggregations of monthly reports withadditional narrative and uploaded financialadditional narrative and uploaded financialreportsreports





GeertGeert BouckaertBouckaert (1993) argues that in order (1993) argues that in orderto be effective performance measurementto be effective performance measurementsystems need to meet the followingsystems need to meet the followingstandards:standards: Validity: sound, convincing and tellingValidity: sound, convincing and telling Legitimacy: measures are accepted from theLegitimacy: measures are accepted from the

bottom upbottom up Functionality: measures contribute to theFunctionality: measures contribute to the

maintenance or development of an organizationmaintenance or development of an organization We would add utility as wellWe would add utility as well


Formative EvaluationFormative Evaluation

PIMS site solicits ongoing feedbackPIMS site solicits ongoing feedback Periodic web-based survey of all usersPeriodic web-based survey of all users Field visits for feedback and promotionField visits for feedback and promotion Semi-structured interviews with Ministry,Semi-structured interviews with Ministry,

Public Health Unit (PHU) and TransferPublic Health Unit (PHU) and TransferPayment Agency (TPA) usersPayment Agency (TPA) users

Evaluation conducted early in the PIMSEvaluation conducted early in the PIMSdevelopment processdevelopment process


MethodsMethods

Feedback interviews began September 2007Feedback interviews began September 2007 22 interviews completed22 interviews completed Survey disseminated October 2007Survey disseminated October 2007 69 survey respondents of 138 potential69 survey respondents of 138 potential

respondents (50% response rate)respondents (50% response rate)


When asked are the indicators in PIMSWhen asked are the indicators in PIMSclear, 55% found the indicators clear, 33.3%clear, 55% found the indicators clear, 33.3%found the indicators unclear, and 11.7%found the indicators unclear, and 11.7%found the indicators very unclear.found the indicators very unclear.


As a tobacco control manager noted:As a tobacco control manager noted:Unfortunately a lot of the, of the informationUnfortunately a lot of the, of the informationon PIMS is, I want to say on PIMS is, I want to say ““bean countingbean counting””and I think thereand I think there’’s some need fors some need forimprovement in that area, because I thinkimprovement in that area, because I thinksome of that is useless information, it doesnsome of that is useless information, it doesn’’ttmatter how many pamphlets we handed out,matter how many pamphlets we handed out,you know, and ityou know, and it’’s hard when yous hard when you’’re in there in thethrow of things to keep track of those kind ofthrow of things to keep track of those kind ofthings. I think therethings. I think there’’s a lot, in the narratives a lot, in the narrative..


When asked to rate the contribution of PIMSWhen asked to rate the contribution of PIMSas a method of reporting to MHP in terms ofas a method of reporting to MHP in terms ofits efficiency:its efficiency:

11.7% found PIMS very efficient11.7% found PIMS very efficient 53.3% found PIMS efficient 53.3% found PIMS efficient 26.7% found PIMS inefficient26.7% found PIMS inefficient 8.3% found PIMS very inefficient8.3% found PIMS very inefficient


When asked to rate the contribution ofWhen asked to rate the contribution ofPIMS as a method of reporting to MHP:PIMS as a method of reporting to MHP:

18.3% found PIMS very helpful18.3% found PIMS very helpful 55% found PIMS helpful 55% found PIMS helpful 26.7% found PIMS not helpful 26.7% found PIMS not helpful


As one interview participant noted:As one interview participant noted: They [MHP] have the information on aThey [MHP] have the information on a

more regular basis, so I would say that theymore regular basis, so I would say that theyhave, faster knowledge I guess, or morehave, faster knowledge I guess, or moreimmediate knowledgeimmediate knowledge..


Another participant observed:Another participant observed:I think itI think it’’s probably a truer accountability,s probably a truer accountability,not that I was, you know, not being clearnot that I was, you know, not being clearbefore but I think itbefore but I think it’’s just, well its just, well it’’s a shorters a shorteramount of time itamount of time it’’s a clearer, clearer datas a clearer, clearer dataso I think it just makes it a betterso I think it just makes it a betteraccountability.accountability.


Asked, to what extend has PIMS helped yourAsked, to what extend has PIMS helped yourorganization track progress in accomplishingorganization track progress in accomplishingworkplanworkplan targets: targets:

36.7% found PIMS useful 36.7% found PIMS useful 48.3% found PIMS not useful 48.3% found PIMS not useful 15% found it useless15% found it useless


PIMS is PIMS is ““a very good check and balance fora very good check and balance forus internally here.us internally here.”” PIMS helps in PIMS helps in ““makingmakingsure that we stay focusedsure that we stay focused”” reported a PHU reported a PHUinterviewee.interviewee.


When asked to rate the contribution of PIMSWhen asked to rate the contribution of PIMStowards program planning:towards program planning:

3.3% found PIMS very helpful 3.3% found PIMS very helpful 43.3% found PIMS helpful43.3% found PIMS helpful 46.7% found PIMS not helpful46.7% found PIMS not helpful 6.7% found PIMS to be useless6.7% found PIMS to be useless


As one interview participants commented:As one interview participants commented: I think it [PIMS] is a helpful tool and I thinkI think it [PIMS] is a helpful tool and I think

yes, it has affected our decision making inyes, it has affected our decision making interms of, where we put our resources, howterms of, where we put our resources, howmuch wemuch we’’re you know the divide perhapsre you know the divide perhapsbetween advocacy, between prevention,between advocacy, between prevention,cessation, those types of issues I think itcessation, those types of issues I think itdoes, it does make a differencedoes, it does make a difference..


Another interviewee noted that PIMS Another interviewee noted that PIMS ““forcesforcesyou to reflect, forces you to provide thatyou to reflect, forces you to provide thattime.time.”” Suggesting that the process of Suggesting that the process ofproviding data to PIMS is beneficial inproviding data to PIMS is beneficial inprogram planning.program planning.


Early Ministry FeedbackEarly Ministry Feedback

System is intuitive, easy to use. For example,System is intuitive, easy to use. For example,Ministry review comments made directly to workMinistry review comments made directly to workplans and users automatically notified.plans and users automatically notified.

Logic of programs and fit within the SFO StrategyLogic of programs and fit within the SFO Strategyis clearly defined at the program level.is clearly defined at the program level.

Automated centralized file management is anAutomated centralized file management is anasset.asset.


Early Ministry FeedbackEarly Ministry Feedback

Work planning in PIMS is a departure fromWork planning in PIMS is a departure fromconventional work planning; difficult to discernconventional work planning; difficult to discernobjectives/priorities of program activitiesobjectives/priorities of program activities

Explanatory text boxes provided for indicators,Explanatory text boxes provided for indicators,but often not usedbut often not used

Indicators are numerous and require adjustmentIndicators are numerous and require adjustmentto better reflect critical activities, outputs andto better reflect critical activities, outputs andassociated outcomesassociated outcomes

Essential to provide training and on-goingEssential to provide training and on-goingtechnical assistance to both Ministry and TPAtechnical assistance to both Ministry and TPAstaffstaff


Preliminary Evaluation ResultsPreliminary Evaluation Results

Overall compliance with work planning andOverall compliance with work planning andon-going reporting in PIMSon-going reporting in PIMS

Data collected through PIMS largely unusedData collected through PIMS largely unusedby program staff/management as reportingby program staff/management as reportingfunctionality is very newfunctionality is very new


System ImprovementsSystem Improvements

New work plan format for this fiscalNew work plan format for this fiscal Increased search functionalityIncreased search functionality Indicator refinementsIndicator refinements


ConclusionConclusion The development of performance measurement systems isThe development of performance measurement systems is

an ongoing iterative processan ongoing iterative process In terms of validity, the performance measures in PIMSIn terms of validity, the performance measures in PIMS

require refinementrequire refinement Legitimacy has been established for all Legitimacy has been established for all TPAsTPAs by a by a

collaborative process of developing performancecollaborative process of developing performancemeasures. For PHUs this process could be moremeasures. For PHUs this process could be morecollaborative.collaborative.

Functionality for work planning and tracking work planFunctionality for work planning and tracking work plantargets requires improvementtargets requires improvement

And, utilization cannot yet be ascertained as the reportingAnd, utilization cannot yet be ascertained as the reportingfunctionality was rolled out after the evaluationfunctionality was rolled out after the evaluation


Contact InformationContact Information

Gala ArhGala ArhOntario Tobacco Research UnitOntario Tobacco Research Unit(416) 978-8306(416) [email protected]@utoronto.ca

Robert SchwartzRobert SchwartzOntario Tobacco Research UnitOntario Tobacco Research Unit(416) 978-3901(416) [email protected]@utoronto.ca

Documents

Formative Evaluation of the Performance Indicators ... · to the Ministry and the field ... entry, data importing and report generation training. Ontario Tobacco Research Unit