Peer Assessment of Public Health Emergency … Assessment of Public Health Emergency Response ... B. Requestor Job Action Sheet ... Identification of why something went wrong and the

PEER ASSESSMENT TOOLKIT

1

www.hsph.harvard.edu/preparedness

Peer Assessment of Public Health Emergency Response

Toolkit


2

Table of Contents

1. INTRODUCTION . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3

WhatisPeerAssessment? PeerAssessmentRationaleIntendedUsersofthisToolkit

GlossaryofTerms

2. THE PEER ASSESSMENT PROCESS . . . . . . . . . .. . . .. . . . . ... . . . . . 8

PeerAssessmentMajorComponents InitiatingaPeerAssessment MasterActivitiesList ConductingaPeerAssessment

3. FACILITATED LOOKBACK: GOING DEEPER . . . . . . . .. . . . . . 12

4. FOLLOW‐UP AND ANALYSIS REPORT , . . . . . . . .. . . . . . . . . . . . 22

5. CASE STUDY: TEXAS WEST NILE VIRUS . . . . . . . . .. . . .. . . . . 25

APPENDICES

A.EvidenceforPeerAssessmentandReferencesB.RequestorJobActionSheet

C.AssessmentTeamJobActionSheetD.SampleMeetingInvitation E.ProjectFunding/AdditionalResources

Rachael Piltch-Loeb, Christopher Nelson, John Kraemer, Elena Savoia, and Michael Stoto, January 2014


3

What is Peer Assessment?

Peerassessmentisaprocessdesignedtoanalyzeapublichealthsystem’sresponsetoanemergency,identifyrootcausesofsuccessesandfailures,andhighlightlessonsthatcanbeinstitutionalizedbytherespondingpublichealthsystemandotherstoimprovefutureresponses.Afteranagencyorgroupofagenciesrespondtoanincidentthatstressestheabilitiesofthepublichealthsystem,thejurisdictionsthatrespondedtotheincidentinitiatesapeerassessmentasa“requestor.”Thisnotonlyprovidesdirectbenefitstotherequestor,whichwillhavetheassistanceofa“peerassessmentteam”intheafteractionreviewprocess,butalsofosterscommunicationandcollaborationacrossjurisdictions,allowingrequestorstoengagewiththeirassessmentteam.Theremaybeindirectbenefitstotheassessmentteamaswell,giventheopportunitytolearnfromthepublichealthresponseoftherequestingjurisdiction.Peerassessmentisnotasubstitutefora“hotwash”immediatelyafteranincident,ratheritservesadifferentpurpose.Theprimarygoalofahotwashorimmediatereviewistoidentifyissuesthatrequireattentionandrecordthefactsaboutwhathappenedbeforememoriesfade.Ratherthanasking“what”and“howmany”questions,apeerassessmentprocessisintendedtohelpjurisdictionsunderstand“how”and“why”problemsoccurredasasteptowardsidentifyingandaddressingcontributingfactorsthatarelikelytobeaproblemisfutureincidents.(Althoughsomehotwashesidentify“strengthsandweaknesses”or“thingsthatwentwell/notsowell,”itisusuallynotpossibletosystematicallyaddress“why”questionsintheimmediatelyaftermathoftheincident).Thereportresultingfromthepeerassessmentprocesscanstandaloneasanincidentreport,servingasorcomplementarytoastandardAAR.Inaddition,theincidentreportcanbesharedwithothersthroughacriticalincidentregistry(CIR)forpublichealthemergencypreparedness(Piltch‐Loeb,2013).APHEPCIRisintendedtoprovideadatabaseofincidentreports,allowingforbothsharingwithothersinsimilarcontextsandfacilitatingcross‐caseanalysis.Assessmentteamsshouldbecomposedofpublichealthpractitionersfromjurisdictionsthataresimilarintermsofsize,populationserved,andpublichealthsystemsinplace(asdescribedindetailedbelow).Peerassessmentteamswillideallyincludethreemembers:onetofacilitatediscussions,anothertotakenotes,andathirdpersontofocusontherootcauseanalysisprocess.

1 Introduction


4

Peer Assessment Rationale Majorpublichealthemergenciesarerelativelyrare,andwhentheydooccur,theydifferinimportantways.Whilewecanbegratefulthattheharmthatemergenciescauseisuncommon,theirinfrequencyhascauseddifficultiesinlearningfromreal‐worldincidentsimpedingsystemsimprovementeffortsinpublichealthemergencypreparedness(PHEP).Thepeerassessmentapproachdescribedinthistoolkitprovidesanopportunityforhealthdepartmentstocollaborateintheireffortstolearnfromsuchincidents.Thegoalsaretoimprovefutureresponsesforthepublichealthagenciesthatrespondedtotheincident,aswellastoidentifybestpracticesforotherhealthdepartmentsrespondingtosimilarincidentsinthefuture.Afteranagencyorgroupofagenciesrespondtoanincidentthatstressestheabilitiesofthepublichealthsystemtorespond,thepublichealthpractitionerorgroupofpractitionersrepresentingthejurisdictionsthatrespondedtotheincidentcaninitiateapeerassessmentprocessasa“requestor.”Thisnotonlyprovidesdirectbenefitstotherequestor,whichwillhavetheassistanceofa“peerassessmentteam”intheafteractionreviewprocess,butalsofosterscommunicationandcollaborationacrossjurisdictions,allowingrequestorstoengagewitheachotherandwiththeirassessmentteam.Theremaybeindirectbenefitstotheassessmentteamaswell,giventheopportunitytolearnfromthepublichealthresponseoftherequestingjurisdiction.Theincidentreportresultingfromthepeerassessmentprocesscanalsoserveasasupplementtoastandardafteractionreport(AAR).Inaddition,thereportcanbesharedwithothersthroughacriticalincidentregistry(CIR)forpublichealthemergencypreparedness(Piltch‐Loeb,2013).APHEPCIRisintendedtoprovideadatabaseofincidentreports,allowingforbothsharingwithothersinsimilarcontextsandfacilitatingcross‐caseanalysis.Thispeerassessmentprocesswasfieldtestedintwojurisdictionsthathaveexperiencedapublichealthemergency:aSalmonellaincidentinAlamosaCounty,Colorado,andamajorWestNilevirusoutbreakintheDallas‐FortWorthMetroplexinTexas.Asitevisitwasconductedforeachincident,duringwhichapeerassessmentteam(agroupofpublichealthpractitionersfromotherjurisdictions)ledpractitionersfromtherespondingjurisdictionthroughadocumentreviewand“facilitatedlook‐back”process(Aledort,2006)toperformarootcauseanalysis.Theseanalysesaredocumentedindetail.TheWestNileVirusexamplecanbefoundintheexampleboxesthroughoutthistoolkitandbothcasescanbefoundinfullintheappendices.Peerassessmentforpublichealthemergencyincidentsenablespractitionerstolearnfromexperiencewhichmovespublichealthemergencypreparednessintothecultureofqualityimprovement(QI)recommendedbytheNationalHealthSecurityStrategy(NHSS)andenhancesthehealthsecurityofournation(DHHS,2009).ContinuousQIisalso


5

fundamentaltothePublicHealthAccreditationBoard’snationalaccreditationprocess.Throughthepeerassessmentprocess,publichealthpractitionersassisttheirpeersinassessingthesuccessesandfailuresinvolvedinrespondingtoapublichealthemergencyandworkwiththemtofindsolutions.Researchhasshownthatstandardqualityimprovementmethodssuchas“learningcollaboratives”maynotbeappropriateinthecontextofPHEP.Thisisduetothelackofevidencebasedandagreeduponperformancemeasures,andthedifficultyofcarryingoutrapidplan‐do‐study‐act(PDSA)cyclesandmeasuringprocessesandresultsafterrareevents(Stoto,2013a).Thepeerassessmentprocessisdesignedtohighlighttherootcausesofthesesuccessesandfailures,andleadtothoughtfullessonslearnedandimprovementstrategiesthatcanbeinstitutionalized.Forfurtherinformationontheresearchsupportforthisapproach,seePartIIIofthisreport.

Intended Users of this Toolkit Theprimaryplayersinvolvedinapeerassessmentaretherequestor,thepublichealthpractitionerorgroupofpractitionersrepresentingthejurisdictionsthatrespondedtotheincident,andtheassessmentteamorassessors,thepeerpublichealthpractitionerswhohavebeencalledupontoreviewtheresponsetotheincident.Anidealpeerassessmentteamwillconsistofameetingfacilitator,anotetaker,andperhapsathirdindividualtohelpwiththerootcauseanalysis.Inthistoolkitthereareseparateguidesforrequestorsandassessors.Thistoolkitbeginswithabriefintroductiontothepeerassessmentprocessandaglossaryoftermsused.Thisisfollowedbydetailedinformationforbothrequestorsandassessmentteamsaboutthepeerassessmentprocessandrootcauseanalysis,illustratedwithanexamplebasedontheDallas‐FortWorthMetroplexWestNileVirusoutbreak.Followingthis,twoseparateandparallelsectionsprovidedetailedJobActionSheetsforbothrequestorsandassessmentteams.Thefinalsectionsummarizestheresearchbackgroundsupportingthepeerassessmentprocess.Toillustratethepeerassessmentapproachinmoredetail,twocompleteincidentreportsareincludedasappendices,oneistheexampleusedthroughoutthetoolkit,theTexasWestNilevirusexperience,andtheotheristheSalmonellaoutbreakinAlamosa,Colorado.Afinalappendixpresentstemplatesthatcanbeadaptedtoemploythepeerassessmentjurisdictioninothersettings.ThedocumenthasbeendevelopedbyaresearchteambasedatGeorgetownUniversity,withfundingsupportawardedtotheHarvardSchoolofPublicHealthPreparednessandEmergencyResponseResearchCenterunderacooperativeagreementwiththeCentersforDiseaseControlandPrevention(CDC)grantnumber5P01TP000307‐04.


6

GLOSSARY OF TERMS Adaptation:AchangemadeduringtheresponsetoanincidentthatspecificallylimitedtheresponsechallengeinthatsituationAssessmentteam(alsoseefacilitator):PeerpublichealthpractitionerconductingthereviewoftheresponsetoanincidentattherequestoftherespondingjurisdictionCriticalIncidentRegistry(CIR):Acatalogueofreportsonajurisdiction’sresponsetoanincident.Theregistryisindevelopmentinconjunctionwiththeothermethodsdiscussedinthisdocument,especiallypeerassessment.ContributingFactor:Underlyingfactors(modifiable&un‐modifiable)thatleadtotheimmediatecauseFacilitatedLookback:Amethodtobringindividualstogethertodiscussaparticularincidentthatfollowsa“systemsimprovement”spirittogetatcausesthatcontributedtotheincident(Aledort,2006)ImmediateCause:Initiallyexplicitreason(s)forresponsechallengesthataffectedmeetingtheresponseobjectiveLessonforSystemsImprovement:IdentificationofwhysomethingwentwrongandthewayinwhichpreventsimilarresponsechallengesinfutureeventsLookbackFacilitator:Peerpublichealthpractitionerconductingthereviewoftheresponsetoanincidentattherequestofaninvolvedjurisdiction.(Facilitatorspecificallyreferstothisperson’srolebeingtoleadthefacilitatedlookbackin‐personmeeting)Objective:ThegoaloftheresponsePeerAssessmentModel:Theengagementofpublichealthpractitionersinanalyzingtheresponseofapublichealthsystemresponsetoaparticularincident.PublicHealthEmergencyPreparedness(PHEP):Thecapabilityofthepublichealthandhealthcaresystems,communities,andindividuals,toprevent,protectagainst,quicklyrespondto,andrecoverfromhealthemergencies,particularlythosewhosescale,timing,orunpredictabilitythreatenstooverwhelmroutinecapabilities.Preparednessinvolvesacoordinatedandcontinuousprocessofplanningandimplementationthatreliesonmeasuringperformanceandtakingcorrectiveaction(Nelson,2007).Requestor:Memberofthejurisdictionwhichhasaskedforapeerassessmentoftheirresponsetoanincident


7

RootCauseAnalysis:Aqualitative,retrospective,qualityimprovementtoolusedtoanalyzeadverseincidentsandsentinelevents(e.g.,apreventableerrorleadingtodeath,seriousphysicalorpsychologicalinjury,orriskofsuchinjury)atthelowestsystemlevel(Wu,2008)ResponseChallenge:ItemthatlimitedtheabilitytorespondtoanelementofthepublichealthemergencyincidentStoryArc:Theoverarchingseriesofeventsthatledtochallengesinmeetinganobjective,includingthevariousfactorsthatenabledorbarredtheobjectivebeingmet.


8

Peer Assessment Major Components Themajorcomponentsofthepeerassessmentprocessareasfollowsandmoredetailintheformofjobactionsheetsisincludedbelow.

Therespondingjurisdictionpreparesaninitialreportforthepeerassessorstoreview

o Supplementaldocumentationisalsosharedtoprovidethepeerassessmentteamwithasmuchbackgroundinformationaspossible

Asitevisitisscheduledforthepeerassessmentteamtoconductin‐personmeetingswithrequestors

Priortothesitevisitwillbeaconferencecalltoidentifythecriticalissuesintheincident

Theassessmentteamwillprepareslidestofacilitatethediscussionsduringthesitevisit

Anon‐sitemeetingwillbeconductedtoanalyzetheresponsetotheincidentthrougharootcauseanalysisprocess

o Thesitevisitmayincludeone‐on‐oneinterviewsandafacilitatedlook‐backmeeting

Afterthesitevisittheassessmentteamwillfollowupwiththerequestorsasneeded Thepeerassessmentteamwilldraftanincidentreportforthereviewofthe

requestorsandfinalrecommendations

Initiating a Peer Assessment

Typically,apeerassessmentshouldbeinitiatedwithinsixmonthsaftertheresponsetoanincidenthasconcluded;however,insomelong‐durationincidentsitmaybepossibleanddesirabletobeginassessmentplanninganddatacollectionsooner.Theassessorsmaybeidentifiedbytherequestorsintherespondingjurisdictionbyselectingpractitionersfromcontactsinotherjurisdictionsorstates.Insomecases,statehealthdepartmentsornationalprofessionalorganizationsmayhelptoidentifyappropriatepeerassessors.Asitevisitforpeerassessmentteamsshouldoccuratanagreedupontimewithinapproximately60daysoftherequestforpeerreview.

EXAMPLE:TexasWestNileVirusoutbreakInTexas,thestatehealthdepartmentcontactedapractitionerintheHoustonareatoassistinreviewingtheresponsetotheWestNileVirusoutbreakintheDallas‐FortWorthMetroplex.Statehealthofficialswereabletousetheirrelationshipstoidentifyapractitionerwhoknewabouttheincidentbutwasnotdirectlyinvolvedintheresponse.

2 The Peer Assessment Process


9

Master Activities List

Timeframe Peer requestor Peer assessment team Assoonaspossiblefollowing

anincidentPrepareaninitialreport

Withinsixmonthsoftheincident

Requestanassessment

Approximately1monthout SchedulesitevisitsChooseadatagatheringapproach(interviews,facilitatedlookback,etc.)Invitemeetingparticipants

Schedulesitevisits

Approximately2weeksout One‐on‐onediscussionwithassessmentteamsDocumentreview

One‐on‐onediscussionwithrequestors

DocumentreviewApproximately1weekout Establishtheattendancelist

SendareminderemailConfirmmeetinglogistics

Dayofmeeting:1hourbeforemeeting

Meetingset‐up Meetingset‐up

Duringmeeting Participate Facilitatediscussion

Dayofmeeting:1houraftermeeting

Discussfeedbackwithassessmentteam

Discussfeedbackwithrequestor

Within1monthaftervisit WriteanalysisreportMakerecommendations

Within2monthsaftervisit Discussfindingswithassessmentteam

Discussfindingswithrequestorsandmodifyifneeded

Open‐ended Follow‐up Follow‐up

Preparing for a Peer Assessment WhatDocumentsShouldBeReviewed?

Thereviewwillusuallyincludethefollowing: Relevantincidentactionplans(tohelpassessmentteamsunderstandresponse

activitiesandidentifyopportunitiesforimprovement). Documentationofestablishedpartnershipsactivatedduringtheincident(toprovide

contextforassessmentteams). Dataonrelevanthealthandsignificantnon‐healthoutcomes,responseactivities,etc. Mediareports Timelineofmajorevents PreliminaryAfterActionReports(AAR)andrelateddocumentation Otherrelevantdocumentsatthediscretionoftherequestingjurisdiction


10

What information should be shared? Theinitialreportfiledbythejurisdictionshouldbethestartingpointforthesitevisit.Peerassessmentteams,shouldbeginbyreviewingthecompletenessandfacevalidityofthisdescriptionandwhethertheviewofwhathappenedchangedsincethereportwaswritten.Providingdocumentstothepeerassessmentteamearlyintheprocessmayreducetheneedforadditionaldatarequests.Thoughpeerassessmentteamswillhavealreadybeengivenaccesstorelevantplans,reviewinghowthisincident/eventevolvedcomparedtoplanningassumptionswillbeimportanttoisolatetheuniquefeaturesoftheincident.Peerassessmentteamsshouldplantoreviewthehistoryofthedepartmentinrespondingtosimilartypesofincidentsorincidentsofadifferentnaturebutwithsimilarcapabilitiesbeingstressed,throughareviewofrelevantdocuments.IfanAARhasbeendraftedbytherequestingjurisdiction,thisAARshouldserveasapointofreference.

How will site visits be scheduled? Therequestingjurisdictionandpeerassessmentteamwillidentifyadatethatworksforbothoftheirschedules.Sitevisitswillideallybeatleastonefulldaysoastomeetwithasmanyinvolvedpersonnelaspossible,eitherindividuallyorinagroup,andtoinvolvedialoguebetweenassessmentteamsandtherequestingjurisdiction.What logistical arrangements will be necessary? Asitevisitfromapeerassessmentteamwillrequireanin‐personmeetingbetweentherequestorsandthoseinvolvedintheresponseandtheassessors.Therequestorswillberesponsibleforidentifyingavenueforthismeetingthatcanaccommodatethemeetingattendeesandprovidinglunchifappropriate.Thesizeofthevenuewillvaryonthemeetingparticipants.Formoreinformationontheidealvenueforthemeetingandmeetingset‐up,pleaseseethesectionsbelow.

EXAMPLE:TexasWestNileVirusoutbreakAnAARthathadbeendraftedforthestatehealthdepartmentservedasthefoundationfortheissuestobediscussedatthemeetingandaninformativedocumenttoreviewtheincidentandkeypersonnelwhorespondedtotheincident.


11

Conducting a Peer Assessment How will the peer assessment team collect information? Thepeerassessmentteamwillcollectthemajorityofthedatafortheirassessmentthroughin‐personinterviewsandafacilitatedlook‐backprocess.Thepeerassessmentteamwillfirstcontactthestaffmemberwhorequestedtheassessmenttointerviewthatindividualandgetintouchwithothersinvolved.Thefollowingapproachescanbeusedintandem,orasthepeerassessmentteamfindsappropriate.Inourexperience,wehavefoundithelpfultoincludethefacilitatedlookbackapproachdescribedbelow.

InterviewsOne‐on‐oneorsmallgroupinterviewswouldusuallybethefirststepinthesitevisitprocess,ortheycouldbeconductedbyphonebeforehand.Interviewswillusuallybetheprimarywayforpeerstofindoutdetailsabouttheincident.Interviewsshouldbeconductedwithkeypersonnel,beginningwiththepersoninchargeofleadingtheresponse.Interviewsshouldbeconductedwithothersinvolvedintheresponseasindicatedbythejurisdiction.Intervieweesshouldbechosentoensurethatallrelevantperspectivesarecovered.Interviewsconsistofastructuredconversationbetweenpeerassessmentteamsandlocalpractitioners.Iftimeislimited,aninformalmeetingwithleadersoftheresponsecanreplaceindividualone‐on‐oneinterviewsinconjunctionwithafacilitatedlookback.

FacilitatedLook‐backSession

Afacilitatedlook‐backisanestablishedmethodforexaminingpublichealthsystems’emergencyresponsecapabilitiesandforconductingacandidsystems‐levelanalysis.Afacilitatedlook‐backbringstogetherasmanypartiesaspossibleinvolvedintheresponseandmaybemoreeffectiveiftimeorresourcesarelimitedforasitevisit.Theindividualsparticipatinginthelook‐backprocesswillhavehaddifferentresponsibilitiesintheresponse,andbringtheirpersonalperspectivestothetable.Individualscanbefromoutsideofthepublichealthdepartmentandwillideallyincludelocalhealthdepartmentrepresentatives,statehealthdepartmentrepresentatives,medicalpersonnel,emergencymanagers,etc.

EXAMPLE:TexasWestNileVirusoutbreakThepeerassessormetwiththestatehealthofficialswhohadrequestedtheassessmentviateleconferencetwoweekspriortothesitevisit.TheteleconferenceconsistedofaplanningdiscussionandabriefAARreviewtoestablishthekeyissuestodiscusswithmeetingparticipants.Theteleconferencealsoprovidedanopportunityforthestateofficialstosharewiththepeerassessoranyon‐goingchallengesthatwerecurrentlygoingoninthearea.


12

Throughtheuseofaneutralfacilitator,i.e.thepeerassessmentteam,decision‐makingcanbeprobedthroughindetailindiscussionswithpublichealthleadersandkeystaffaswellasavarietyofcommunitystakeholderswhowereinvolvedwiththeresponse.Formoredetailonthefacilitatedlook‐backmethodology,seeAledort(2006).

Facilitated Look-back Attendees Alloftheindividualsandorganizationsthatplayedkeyrolesintheresponseshouldbeincludedorrepresented,aswellasparticipantswithvaryingperspectives.Suggestedmeetingparticipantsinclude:

o Leadersoftheemergencyresponsefromrespondingjurisdictionso Emergencyresponderso Representativesfromthelocaljurisdictionsoutsideofexplicitlypublichealth

activities(environmental,publicworks,etc.),dependingonthetypeofincident

o Statehealthworkersinvolvedintheresponseo Volunteercoordinatorso Hospitalandotherhealthcaredeliverysystemrepresentativeso Coalitionleaderso Governmentworkersinvolvedinemergencydeclarationactivities

Ifafacilitatedlook‐backisplanned,thepeerassessmentteaminconjunctionwithleadersoftherequestingjurisdictionshouldsendanemailinvitationtopersonneltobeincludedinthemeeting(describedabove).Theinvitationshouldinclude:thepurposeofthemeeting,thetimeandlocationofthemeeting,the“systemsimprovement”spiritofthemeeting,andthe“no‐faultzone”attitudethatwillencourageopendiscussion.Afollowupemailshouldbesentpriortothemeetingtoremindparticipants.

EXAMPLE:TexasWestNileVirusoutbreakThefacilitatedlookbackwasthefoundationofthesitevisitinTexas.DuringthefacilitatedlookbacktherewasrepresentationfromthreedifferentcountieswhohadbeeninvolvedintheWNVresponseaswellasstateregionalofficerepresentatives.

3 Facilitated Lookback: Going Deeper


13

Meeting Logistics Thelocationofthemeetingwillbedeterminedbytherequestingjurisdictionbasedonaccesstoameetingspace;however,themeetingspaceshouldincludePowerPointdisplay,boardsorwritingpadsthatcanbewrittenonduringthediscussion,andaroundtablesetupthatcanaccommodate20‐25people(willvarybasedonmeetingattendance).

Chairsintheroomshouldbeorganizedsoparticipantscanfaceeachother,andthefacilitatorwillideallybetheonlyindividualwhostandsduringthediscussion.

Tentcardsshouldbeon‐siteforparticipants.Tentcardsshouldthenbedisplayedat

eachindividual’sseat.How will the peer assessment meeting be facilitated? Thepeerassessmentteamwillfacilitatethemeeting.Therequestorswillparticipateinthismeetingwiththeircolleaguesandpeersfromotherrespondingjurisdictions.Therequestorsshouldplantosharetheirthoughtsontheresponse.Apreviewofthemeeting’sstructureislistedbelowandcanbefoundinmoredetailinthesampleslidedeckinAppendix3:

Agenda

Objectivesformeeting

Corequestionstobediscussedtoday

Groundrulesfordiscussion

Introductionofparticipants

Overviewofrootcauseanalysis,toexploreissuesfordiscussion

Rootcauseanalysisdiagramexplanation(seesectionbelowforfurtherdetails)

o Explainthisisaprocesstogetatrootcausessoastolimitthesameissues

occurringinthefutureandbetterlearn

o Anexplanationoftherootcauseanalysisprocessdemonstratedthroughan

examplefromtheTexasWestNileVirusoutbreak.

Rootcauseanalysisdiagramexamplefrompeerassessmentteam’sexperience

RootcauseanalysismethodsinPHEPexplanation

Reviewoftimelineofevents:

o Probewhicheventsweretriggersforaction


14

Figuresorvisualsrepresentingevent

Discussionoffirstissue(capabilityoridentifiedtopic):

o Considerthefollowing:

Whowasinvolved

Whywasthisanissue

Whatfactorscontributedtothisbeinganissue

Whatfactorswereinthedepartment’scontrol,whatfactorswere

unmodifiablebutplayedarole

o Duringthediscussion,theindividualtaskedwithobservingrootcausesshould

attempttofillintheRCAdiagram

RCAdiagramshouldthenbedisplayedonthescreenforfeedback

Discussionofsecondissue(repeatforasmanyissuesastimefor)

LessonsLearnedfromdiscussions

Actionplanforthejurisdiction

Feedbackforpeerassessmentteam

Contactinformationoftheassessmentteamtoprovideanyadditionalcomments

Ifpossible,afterthemeetingthepeerassessmentteam(s)shouldmeetwiththeleaderswhorequestedtheassessmenttoaskforanyadditionalcommentsandhowtheybelievethemeetingwent.

EXAMPLE:TexasWestNileVirusoutbreakPost‐meeting,thepeerassessorandstatehealthofficialsmetforapproximatelyonehourtodiscusshowthemeetingwentandwhatcouldhavegonebetter.ThediscussionincludedhowthestatehealthofficialscouldupdatetheirAARbasedonwhattheyheardduringthemeeting.


15

How Should the Discussion Progress during the Site Visit? Havingpractitionersengagedinthepublichealthresponseparticipatinginanin‐personmeeting,providesanopportunitytodiscussrootcausesofspecificresponsechallengesexperiencedduringtheincident.Thisformatisdifferentthanatypical“hotwash”andisintendedtogetattherootcausesofsuchchallengesratherthaninitialimpressions.Thusthepeerassessmentteamshouldleadthegroupintherootcauseanalysisprocess:

Thefacilitatorwillleadthediscussion.Thisdiscussionshouldbedocumentedbythenote‐takerandserveasthebasisforthereporttobedevelopedbythepeerassessmentteam(s)followingthemeeting.Therequestorsshouldparticipateintheprocessbelowbycontributingtheirknowledgeandexpertise.BelowisafullexampleofthisprocessfromtheTexasWestNileVirusincident..Throughoutthestep‐by‐steprootcauseanalysisprocessdescribedbelow,relevantportionsarehighlightedtodemonstratehowthisoccurredduringthistestcase.


16

Step1:Definethestoryarc

Thepeerassessmentteam,afterreviewingthecontextandincidentdescription,shouldidentifyandoutlinethestoryarc–thesetofmajorissuesandeventsthatsetsthestageforeachresponsechallengethatwasdiscussedatthemeeting.Thestoryarccanberesponseobjectivespecific(seeStep2.).Storyarcstypicallyincludeafewpiecesofcontextualinformationrelevanttoexplainingwhytheobjectivehascomeabout.Thestoryarcshouldmentioneventsthataffectedtheoutcomeoftheresponse.Theseeventsareaseriesof“pivotal”nodes(events,decisions,orpointsintime)thatcouldhavegonedifferentlyleadingtoadifferentoutcome.

Step2:Identifycandidatechallenges

Inaccidentorclinicalincidentinvestigations,identifyingadverseeventsoftenfocusesonfairlyobvious,discreteoccurrences–thepatientdiesorisphysicallyinjured,theairplaneenginefallsoff,etc.Publichealthemergencieswillusuallyinvolveahostofindividual“responsechallenges,”someobvious,othersnot‐so‐obvious.Theseresponsechallengeswillhavebeendiscussedduringthemeeting,butmaybefurtherexploreduponreflection.Theassessmentteammustfirstconsiderwhattheparticipants’perceptionsofwhatwerethemostimportantresponsechallengeswere–i.e.theonesthathadanimpactontheoutcome,oratleasthadthepotentialtoimpacttheoutcome,butalsosubjectthemtocriticalassessment.Toframetheresponsechallenges,itisimportanttoconsiderwhattheresponseneedswere.Indefiningtheresponseneed,thepublichealthsystem’sgoalandobjectivesshouldbedocumented.Whatdidthepublichealthsystemhopetoachieveinrespondingtothisevent(outbreak,weatherincident,etc.)?Otherprobingquestionscouldinclude:

EXAMPLE,Step1.StoryArc:TexasWestNileVirusoutbreak

EXAMPLE,Step2a.Objective:TexasWestNileVirusoutbreak

StoryArc:CountiesinTexassendlabsamplestothecentralstatelabsinAustinorusetheirownlocallabs,dependingonthesizeofthecounty.DallasCountyusedtheirownlabs,whileDentonandTarrantCountysentmosquitosamplestothestate.Samplesandresultsoverwhelmedlabsandwerecollectedindifferentformats,resultingindatasetsthatcouldnotbeeasilycomparedor

understoodandlimitingtheabilitytointerpretorusesurveillancedata.

Objective: Providesituationalawarenesstoguideresponseefforts

RelatedCapabilities: PublicHealthLaboratoryTesting(CDCPHP12)andPublicHealthSurveillanceandEpidemiologicalInvestigation(CDCPHP13)


17

TowhichPHEPcapabilitiesdidthesegoalsrelateto? Atwhatpointintimeweregoalsset? Whatwereseenastheprioritypublichealthfunctionsinrespondingtothe

incident?Step3:Selectasubsetofresponsechallengesforin‐depthanalysis

Basedontheinitialanalysis,theassessmentteammayneedtoselectashortlistofresponsechallengesforfurtheranalysis.Aftertheperceivedgoalsareidentified,trytoidentifywhatpreventedgoalfrombeing(easily)met.Askthequestion:Workingbackwards,whatledthejurisdictiontowardtheresultingoutcome?Selectionmaybeguidedbythefollowingcriteria:

Importancetotheresponseunderanalysis Likelyimportancetofutureresponses Relevancetokeystakeholders Potentialforimprovement

Step4:Assessresponsechallenges

Belowaresomeofthewaysassessmentteamsmayassessresponsechallenges:

Compareperceivedchallengeswiththebroadersetofresponseactivities.Consultingincidentactionplans,timelines,andothersources,theassessmentteammayaskwhetherperceivedchallengesmayhavebeencausedbydeficienciesinresponseactivitiesnotidentifiedbyinformantsatthetimeoftheemergency.Itmaybeusefultoconsultjurisdiction’sresponseplans,thespecificresponsefunctionslistedforrelevantcapabilitiesintheCDC’sPHEPCapabilitiesguidance,theTargetCapabilitiesList,orothers.Wherepossible,actionstakenduringtheresponsecanbecomparedwiththresholds/standards(e.g.,timefor

EXAMPLE,Step4.AssessResponseChallenges:

WNV,TX

EXAMPLE,Step3.ResponseChallengesforin‐depthanalysis:TexasWestNileVirusoutbreak

Thethreepivotalresponsechallengesidentifiedearlyinthediscussionwere:1.Surveillancedata2.Mitigationthroughaerialspraying3.CommunicationthroughatypicalchannelsandwiththepublicTherootcauseanalysisprocesswasrepeatedforeachofthesegeneralresponsechallengesthattaxedthejurisdiction’sabilitytorespond.TheexampleintheseboxesistakenfromtherootcauseanalysisprocessofChallenge1.

TrackinghumanWNVcasesand

deaths

MonitoringWNVinmosquitos


18

activatingtheEOC).Whileitisunlikelythattherewillbeasingle,definitivelistoftasksthatshouldhavebeenexecuted,thisprocesswillatleastprovideacounterweighttoinformantperceptions.

Lookforconvergence/divergenceinstakeholderviewpoints.Generally,ifrepresentativesfromrespondingagenciesfrommost/allpartsofthesystemshareacommonperceptionofaresponsechallenge,itshouldgiveusmoreconfidenceinitsveracity.However,itisimportanttonotedivergencesinperceivedchallengesandseektoassesswhetherthosearesystematicallyrelatedtorank,placeintheorganization,trainingbackground,etc.o Onwhichresponsechallengeswastherebroadagreementamong

meetingparticipants?o Onwhichresponsechallengeswastheredisagreementamongmeeting

participants?Weredisagreementsrelatedtopositionwithintheorganization,professionalbackground,orotherobservablefactors?

Examinewhat‐ifanalysis.Insomecasesitmaybepossibletoobservewhatwouldhavehappenedunderanotherresponse(e.g.,PODsthatoperateusingslightlydifferentprocedures),buttheseopportunitiesarerare.However,itisoftenpossibletousedocumentsandinformantdiscussionstoexplore“what‐if”scenariosasawaytoassessingtheextenttowhichaperceivedchallengeactuallyaffectedoutcomes.

Consideradaptationsandsolutions.Tomeetaresponsechallenge,thereareoftenadhocchangestoproceduresornewwaysofajurisdictionrespondingthathappenduringanincident.Theseon‐the‐spotchangesinproceduresmayenablethejurisdictiontobetterrespondtotheincidentandthereforemitigatetheresponsechallengethatisoccurring.Theseadaptations,orchangesmadeduringtheresponsetoanincidentthatspecificallylimitedtheresponsechallengeinthatsituation,orsolutions,awayofsolvingtheongoingproblemshouldbedocumentedforthereferenceofthemeetingparticipantssoastobeconsideredforalessonlearnedorincorporationintofutureplanningefforts.Toidentifyifanadaptationorsolutionwasutilizedforthisresponsechallenge,consider:o Wastheresponsechallengelimitedbyanactionthejurisdictionor

anotherorganizationtook?Ifso,whatwasthisaction?o Despitethechallenge,wasthejurisdictionsuccessfulinmeetingits

objective?Whatactivitiesenabledthejurisdictiontodoso?o Whatwouldhavehelpedthejurisdictiontobettermeettheresponse

challenge?

EXAMPLE,Step4.AdaptationsandSolutions:TexasWestNile

Virusoutbreak

AdHocmechanismstosharedataelectronically


19

o Wereanystepstakentowardsthiswhiletheincidentwasoccurring?Ifso,whatwerethosesteps?

o Didanewgrouptakeonresponsibilitiesnottypical?Whotookactiontowardsanadaptationorsolution?

Step5:Identifyfactorsthatcontributedtotheresponse

5a.ImmediateCauses:Foreachresponsechallenge,usethedatacollectedtosolidifytheimmediatecauses.Thesewerethefirstexplanationsforwhyaresponsechallengeoccurred.Belowisalistofprobingquestionsthatmayleadtofindinganimmediatecause: Whatdecision‐makingandorganizational

factorsinfluencedtheresponseactionsundertakentoaddressspecificresponsechallenges?

Whathumanfactors(staffactions,trainingandexpertise)influencedtheresponseactionsundertakentoaddressspecificresponsechallenges?

Whatpopulationfactors(demographics,hazardsvulnerability,etc.)influencedtheresponseactionsundertakentoaddressspecificresponsechallenges?

Step5b:ExaminethecontributingfactorsMostlikely,foreachimmediatecausethereweremultiplereasonsthatcauseoccurred. Typesoffactors.Thesefactorsmayhave

beenmodifiable,un‐modifiable,orpre‐determined.Modifiablefactorscanbechangedbytherespondingjurisdiction,un‐modifiablecausesareoutofthecontrolofthejurisdictions,andpre‐determinedfactorswerepartofanexistingplanorstructurebuttechnicallyinthecontrolofthejurisdiction.Thesamefactorcaninfluencemorethanoneimmediatecause.Thefactorsdescribedinthereportshoulddrawonthosediscussedduringthemeeting,andgoonestepfurther,

EXAMPLE,Step5a.ImmediateCauses:TexasWestNileVirus

outbreak

Inconsistent results in state, local, and private labs

Delays and inconsistencies in transmitting data throughout the public health system

Communicating through unfamiliar “emergency” rather

than standard channels

EXAMPLE,Step5b.ContributingFactors:TexasWestNileVirus

outbreakDifferenttestingstandardsin

state,local,andprivatelabs

Labcapacityandnatureoftests(PCRvs.viralculture)

Multipledatasystems

Lowerpublichealthcapacitiesinruralcounties

Routineprotocolsdidnotworkathighlevelsofthroughput/surge


20

soastodefinetherelationshipofthesefactorstotheimmediatecauses. DOTMLPF.Toidentifywhattriggeredthejurisdiction’sdecision‐making,

considertheDoctrine,Organization,Training,Materiel,Leadership,Personnel,Facilities(DOTMLPF)frameworkdevelopedinthemilitary:

o Doctrine:Plans,policies,protocols&proceduresdevelopedandimplementedbeforeaneventthatdetermineresponsesduringtheevent;includes“cultural”differences,informalnorms/expectations

o Organization:howweorganizetorespondtoanemergency,e.g.federal/state/localroles,regionalresponseorganizations

o Training:howwepreparetorespond;basictrainingtoadvancedindividualtraining,varioustypesofunittraining,jointexercises,etc.

o Materiel:allthe“stuff”necessarytoforaneffectiveresponseo Leadershipandeducation:howweprepareleadersatvariouslevelsand

indifferentorganizationstoleadtheresponse;professionaldevelopmento Personnel:availabilityofqualifiedpeopleforemergencyresponse

operationso Facilities:realproperty

Noteso Whileitmaynotsurfaceeveryissue,theprocessofdeepanalysis

describedhereislikelytoidentifymoreissuesthanamoresuperficialanalysis.Forinstance,issuesthatappearatfirsttobepersonalconflictsbetweendifferentpeoplewhowereinvolvedwiththeresponseoftenreflectdeepersystemsissuesthatareeasierandmoreusefultotalkabout

o Sometimesthepointsthatcomeoutinsuchananalysismaybesosensitivethattheycannotbeinapublicreport,butatleastthereissomelearningfortherequestingjurisdiction.Inothercasesitmightbepossibletodescribetheissueinsufficientlygeneraltermsthatitcanbeincludedinapublicreport.

Step6:Documentlessonsforsystemsimprovement

Lessonslearnedareconclusionsthatemergefromtherootcauseanalysestobeconsideredwhenmakingfutureplansorrespondingtoasimilarincidentinthefuture.Lessonslearnedmayaddressorganizationalissuesorprocessissues,butshouldconsiderthecontributingfactorsandwhatcanbedonetomitigatesuchresponsechallengesinthefuture.Thefacilitatorshouldprobethegroupforlessonslearnedthatcometomindimmediatelyafterthediscussion;however,additionallessonslearnedmaybeidentifiedthroughbrainstormingafterthecompletionofthesitevisit.Lessonslearnedmayalsoberevisedasotherresponse

EXAMPLE,Step6.LessonsforSystemsImprovement:TexasWestNileVirusoutbreak

Lessonsforsystemimprovement:

• Needforclear,comprehensive,uniformdatasystems

• Needforacentralincidentcommandsystemineachcountythatthencoordinateswiththestateregionalofficeandwiththeothercounties


21

challengesarediscussed.Thefinalizedlessonslearnedwillbeincludedinthefinalanalysisreport(seebelow).


22

What Kind of Follow-up Can Be Expected after the Site Visit? Afterthesitevisit,theassessmentteamwillsummarizewhatwassaidatthemeetinginananalysisreporttobesharedwiththerequestorsandothermeetingparticipants.Thereportisdesignedtosharefindingswiththoseinvolved,getrequestors’input,andrespondtoanyconcernsthatmayberaisedbytheassessors.Thereportcontainsinformationgatheredthroughthefacilitatedlook‐backand/orinterviews.Theassessmentteamswillthenprepareafinalreport.Thissectionprovidesinformationforreviewingthematerialcollectedatasitevisit,completinganincidentreportforaCIRentry,andcommunicatingwiththepeerassessmentteamsfollowingthesitevisitprocess.Howshouldanincidentreportbeorganized?WhethertheincidentreportissolelyfortherequestingjurisdictionortobesubmittedtothePHEPCIR,thereportshouldincludefourcomponents:abriefsummaryoftheincidentincludingthePHEPcapabilitiestestedduringtheincident;abackgroundsectionthatincludesprevioushistoryofthehealthdepartmentinrespondingtosimilartypesofincidents;adescriptionoftheincidentthatincludesinvolvedparties;andaninternalassessmentoftheresponsetotheincident.SampleincidentreportscanbefoundinAppendices1and2.Theinitialreportwillbedesignedtoinformanappropriatepeerassessmentteamofthetypeofincidentthatoccurred,department’sstaff,andothersinvolved.WhatWilltheAnalysisReportInclude?

Abstract.AshortsummaryoftheCIRentry,includingabriefdescriptionoftheincident,capabilitiesanalyzed,andkeyfindings.IftheCIRentryintentionallyleavesoutimportantcapabilities(i.e.,tolimitthescopeofthewrite‐up)thisshouldbeexplicitlynoted.

Context:Basedonwhatthepeerassessmentteamreadpriortothesitevisit,andwhatwaslearnedduringthesitevisit,thereportshouldstartwithadescriptionofthejurisdiction.Thiscontextshouldincludeanyinformationneededtounderstandingtheanalysisandfindings,andmayincludethejurisdictionalenvironmentthehealthdepartmentoperatesin(home‐rule,bystate,etc.),thenumberofjurisdictionsinvolved,informationaboutthelocationofthehealthdepartment,populationserved,andleadershipofotheragenciesthatwereinvolved(i.e.publicworks,emergencyresponders,agriculture,etc.),plansandplanningprocesses,etc.

4 Follow-up and Analysis Report


23

IncidentDescription:Thisshouldbeastraightfactpatternofwhathappenedduringtheincidentthatwasdiscussedduringthesitevisit.Thepeerassessmentteamwillusetheirdiscretionwhendecidingwhatinformationtoinclude.

Analysis:theanalysissectionemploysroot‐causeorsimilaranalysistounderstandwhetherandwhyparticularPHEPcapabilitiesweresuccessfullyemployedinresponsetotheincident(Piltch‐Loeb,2013).ThisapproachisdiscussedinAppendix3(template)andintheanalysissectionabove.Formanyoftheissuesdiscussedduringthesitevisit,therewillbesimilarrootcauses.Thebelowprocessisanotherdescriptionoftherootcauseanalysisprocess,tocomplementwhatisdiscussedinpersonwiththejurisdictionandservesasanotherpointofreference(inadditiontotheslidesintheappendix):

Whatifanyrecommendationswillcomefromthereport?Basedontheidentifiedrootcauses,peerassessmentteamsshouldmakerecommendationsforchangingprotocolsandprocessesforthefuture.Theserecommendationsshouldbeinlinewithwhatwasdiscussedattheon‐sitemeeting,andcanalsoincludethepeerassessmentteamsownexpertise.Theserecommendationsshouldbedocumentedtobediscussedwiththerequestingjurisdiction.Recommendationsshouldbeactionableitems.Inmakingtheserecommendations,considertheseelements:

Whatlessonslearned/recommendationscanbegeneralizedusingthestoryarc? Whatactionsmightpreventsimilarweaknessesorbuildonstrengthsinfuture

responses?o Considertheactionsthedepartmentcouldhavetakentocreatedifferent

outcomeso Considerthealternativeactionsthedepartmentcouldhavetaken

Whatchangescouldbeimplementedpriortofutureeventstochangetheoutcomeslateron?

o UnliketypicalAAR/IPs,peerassessmentreportsdonotneedtoincludeeverycorrectiveactionidentified.Rather,sincethepurposeistoidentifyissuesandpotentialsolutionsthatmightberelevantinothersituations,solutionsthatseeminnovativeandpotentiallygeneralizabletosimilarjurisdictionsshouldbeincluded.

Shouldtheresponsegoalsbedifferent? Indifferentcircumstances,wouldthisresponsehavebeenappropriate?

Whatdothelessonslearnedtellusabouttheunderlyingcapacityofthesystemto

respondwhentheabovementionedcapabilitiesarestressed?


24

WhatSortofFollow‐upDiscussionShouldOccurtoEnsureAccuracy?Therequestorwillreceivethepeerassessmentteam’sreport,whichwillincludeadescriptionoftheincidentandtherootcauseanalysisreport.Therequestorshouldreviewthisreportandidentifyanyissuesthatdonotappeartoaccuratelyreflectwhatwasdiscussedoranyrecommendationsthatdonotseemappropriateforthejurisdiction.Afollowupmeetingviaconferencecallwillbescheduledtodiscusstheseconcernsandtheprocessforimplementinganyrecommendations.Aformalprocessmaybeneededfortherequestingorganizationto“approve”thepeerassessmentteam’sreportandokitfordistributiontoregistryoroutsidepartners.Thisshouldbeanagreedupondeliverableattheinitialrequestandagreementtoconductapeerassessment.

WillThereBeAdditionalFollow‐up?Additionalfollowupmayberequestedbythejurisdiction,CIRadministrators,orpeerassessmentteamsattheparties’discretion.

EXAMPLE:TexasWestNileVirusoutbreakThepeerassessorssharedtheirrootcauseanalysisdiagramsandcorrespondingsummarywiththerequestingpractitioners.Therequestorsthencommentedonthediagramsandmadeafewsubstantivesuggestionsbasedontheirperceptionofthemeeting.Theassessorsconsideredthesechangesandafterdiscussingthemviateleconferencewiththerequestorsmadetheappropriateadjustmentstoreconcilethereport.


25

Requestor:TexasStateDepartmentofStateHealthServices(onbehalfoflocalhealthdepartmentsintheDallas‐FortWorthMetroplex)Assessor:HarrisCountyPublicHealth&EnvironmentalServicesAbstract

Inthesummerof2012,theDallas‐FortWorthMetroplexexperiencedasevereWest

NileVirus(WNV)outbreakinwhichmorethan1,868confirmedcasesofWestNilediseaseand89WNV‐relateddeathswerereported.Theincidentstressedanumberofpublichealthpreparednesscapabilitiesincluding

PublicHealthLaboratoryTesting(CDCPHPcapability12)andPublicHealthSurveillanceandEpidemiologicalInvestigation(CDCPHPcapability13),limitingthestateandlocalpublichealthsystem’stoprovidesituationalawarenesstoguideresponseefforts;

Vectorcontrol(CDCPHPcapability11:Non‐PharmaceuticalInterventions),includingaerialsprayingandothermosquitocontrolactivities;

EmergencyPublicInformationandWarning(CDCPHPcapability4)tocommunicatewaysthatresidentscouldusetoprotectthemselves;and

EmergencyOperationsCoordination(CDCPHPcapability3)andInformationSharing(CDCPHPcapability6)tocoordinateeffortsbetweenandamongstateandcountypublichealthandotherpublicandprivateorganizations.

Lessonsforsystemimprovementfromananalysisofthisincidentincludetheneedfor:

clear,comprehensive,mosquito‐centric,uniformdatasystems; acentralincidentcommandsystemineachcountythatthencoordinateswiththe

stateregionalofficeandwiththeothercounties; pre‐determinedscience‐basedtriggersformitigationeffortsaswellasmutual

assistanceplanswithsprayingcompaniesandamongcounties;and ameanstoshareinformationwithotherjurisdictionslocally,withthestate,and

nationally.

Context

AlthoughWestNileViruscaseswerereportedthroughoutTexasin2012,theoutbreakwasespeciallysevereinthethreecountiesintheDallas‐FortWorthMetroplex(Dallas,Tarrant,andDenton),whichtogetherarereferredtoasthe“Metroplex.”WestNilevirusandothermosquito‐bornediseasesarerareintheMetroplex.Itwasthefirsttimein

5 Case Study: Texas West Nile Virus


26

overfortyyearsthatsomemitigationactivities,suchasaerialspraying,hadbeenconsideredorusedasavectorcontrolactivitytopreventdisease.

DallasCounty,thelargestofthethreehasapopulationofapproximately2.4million,TarrantCounty,whichincludesFt.Worth,hasapproximately1.8millionpeople,andDentonCountyhasapproximately600,000peopleandismoreruralthantheothers.Eachcountyincludesdozensofcities.Texascountiesaregovernedbyanelectedcouncilandjudge(countyexecutiveofficer).

DallasCountyhasitsowndepartmentofhealth,whichhasindependentrelationshipswithlaboratories,andoutsidecontractors.TarrantandDentoncountieseachhasitsownpublichealthdepartment,butreliesonthestateforlabsupportandotherresources.Texasisahome‐rulestateinwhichtheTexasDepartmentofStateHealthServices(DSHS)supportslocaljurisdictionsintheirpublichealthdecisions.TherearehundredsofjurisdictionswithinthestateofTexas,eachwiththeirowndecisionmakingstructureandpublichealthdepartmentorservice.Manycountiesdonothavepublichealthdepartments,andDSHShasregionaloffices(includingonefortheMetroplex)tosupportthesejurisdictions.

IncidentDescription

The2012WestNileVirusseasoninTexaswasthemostsevereonrecord.Atotalof1,834humancaseswereconfirmedthroughoutthestatebetweenJuneandDecember17,2012,including836neuroinvasivecasesand86deaths.Manyfactorsinfluencedtheseverityofthisoutbreak,includingtheprolongeddrought,hightemperatures,activemosquitopopulation,andotherenvironmentalissues.Inaddition,manyofthehumancasesoccurredinareasofthestatewherethenumberofWNVcasesistypicallylow,particularlytheDallas‐FortWorthMetroplexwhichsawmorethan1,868confirmedcasesofWestNileneuroinvasivediseaseand89WNV‐relateddeathswerereported.

Duringthespringandearlysummerof2012,theDallas,Tarrant,andDentoncountyhealthdepartmentsandDSHSperformedroutinesurveillanceandepidemiologicactivitiesrelatedtoWNV.DSHSexecutivestaffwerefirstnotifiedofanincreasednumberofcasesofhumanWNVinfectioninJuly.Asaresult,conversationsandplanningactivitieswereinitiatedamongDSHSHealthServiceRegion(HSR)leadership,localhealthdepartmentleadership,emergencymanagementagencies,andlocalelectedofficials.OnceWNVwassuspected,surveillancedatapouredintothestatefromcountiesthroughoutTexas,inavarietyofformats.InsomelocationssuchasDallasCounty,locallabcapacitysupportedrapidpolymerasechainreaction(PCR)testing,butotherlocationsdidnothavethiscapacityandrelievedheavilyonstatelabs.TheDSHSlabroutinelyperformsculturebasedtestingofmosquitosamplesaspartoftheCDCArbovirusMosquitoSurveillanceNetwork.ThestatelabdidbeginPCRtestingofmosquitopoolsoncetheoutbreakwasidentified.DSHSactivatedtheStateMedicalOperationsCenter(SMOC)anditspublichealthemergencypreparednessfunctionsonAugust9,2012,thesamedayDallasCountyJudgeClayJenkinsdeclaredapublichealthemergency.Thissignaledthattheoutbreakhadreachedacriticalmilestoneandthatnormalcontrolandabatementmeasuresatthelocal


27

levelmightnotbeadequatetopreventanincreasingincidenceofdiseaseoravertincreasingnumbersofdeathrelatedtotheneuroinvasiveformofthedisease.Tosupporttheresponsetothisoutbreak,DSHSstaffinAustinandthehealthserviceregionsbeganamulti‐facetedapproachtosupportlocalhealthdepartmentsandelectedofficialstoprevent,mitigate,andrespondtotheoutbreak.DSHSliaisonswereactivatedtocoordinatetheresponsebetweencounties,andDSHSRapidAssessmentTeam(RAT)wasdeployedtotheMetroplexonAugust14tohelpcoordinateandintegrateresponseactivitiesintheregion.

Despitesubstantialpushbackfromcitizensinpartsofthecommunity,twocountieschosetoconductaerialspraying.ItbeganfirstinDallasCountyonAugust16,withtheDSHShavingcontractedwithClarkeAviation,aglobalenvironmentalproductsandservicescompany.DentonCountydeclaredadisasterdeclarationonAugust22,butweatherdelayedsprayinguntilAugust31.TheSMOCwasde‐activatedonSeptember11.

Timeline

July2012

• DSHSleadershipnotifiedofincreasedtestingofmosquitosandtheincreaseinmosquitostestingpositiveforWNV

• DSHSleadershipinitiatesplanningactivitieswithhealthserviceregions,localhealthdepartments,emergencymanagementofficial,andelectedofficialsintheimpactedareasofthestate

August9,2012

• DSHSactivatestheStateMedicalOperationsCenter• DallasCountyJudge,ClayJenkins,declaresapublichealthemergencyinDallasCounty

August14,2012

• ADSHSRapidAssessmentTeam(RAT)deploystotheMetroplexfortwoweekstoserveinaliaisonrole

• Inthisliaisonrole,DSHSisabletohelpcoordinateandintegrateresponseactivitiesinDallasandDentoncounties

August16,2012

• AerialsprayingbeginsinDallasCountyAugust22,2012

• DentonCountyJudge,MaryHorn,declaresadisasterdeclarationinDentonCountyandrequestsaerialspraying

August31,2012

• AerialsprayingbeingsinDentonCountySeptember11,2012

• DSHSStateMedicalOperationsCenterisdeactivated


28

Analysis

OnMay13,2013afacilitatedlookbackmeetingwasconductedinArlington,Texastoreviewthepublichealthsystemresponsetotheincident.DSHSandeachofthethreecountyhealthdepartmentswererepresented,alongwithDallascountyjudgeClayJenkins.ThemeetingwasfacilitatedbyapeerassessorfromHarrisCounty,Texas,withtheassistanceofresearchstafffromGeorgetownUniversity.Followingareviewoftheepidemiologicalfactsandtimeline,threeissueswerediscussedindetail:surveillance,mitigationefforts(includingaerialspraying),andcommunicatingwiththepublic.

Surveillance

InordertotrackthespreadofWestNilevirusthroughthecommunity,andtoprovidesituationalawarenesstoguideresponseefforts(objectives),stateandlocalhealthdepartmentsutilizedreportsofhumanWNVcasesandtheresultsofmosquitotesting.TheresponsechallengesinthisincidentwereinobtainingconsistentandreliabledatatotrackhumanWNVcasesanddeathsandinmonitoringWNVinmosquitopoolstoguidelocalcontrolefforts.

Therewereanumberofimmediatecausesfortheproblemsencountered.First,laboratorytestinginTexasisdoneinacombinationofstate,county,andprivatelaboratories,dependingonthesizeofthecounty,eachoperatedunderadifferenttimeframe.DallasCountywasabletotestalloftheirownsamples,TarrantCountyhadalabwithlimitedprocessingcapacitysosomeoftheirsamplesweresenttothestate,andDentonCountydidnothavetheirowncapacity.Therewerealsodifferencesbetweenproceduresusedfordiagnostictestingofmosquitopools;somelabsusedpolymerasechainreaction(PCR)whileothersusedadifferentmethodtoculturethevirus.Differencesamonglaboratoriesintestingproceduresandstandardsledtodelaysntheresults.Communicatingresultsfromalloftheselaboratoriestotheappropriatelocalhealthdepartmenttooktime,andcommunicationthroughunfamiliaremergencychannelscomplicatedtheprocess.Toaddresstheseproblems,adhocmechanismsweredevelopedduringtheoutbreaktosharesurveillancedataamongthelocalhealthdepartmentsandDSHS–onceanyjurisdictionreceivedareportitwasblaste‐mailedtoalloftheothers(adaptationsandsolutions),butonlylateintheincident.

Avarietyoffactorscontributedtothissituation.Someareinthenatureofsurveillanceandlaboratorytesting,andareunchangeableatleastinthenearterm:differenttestingstandardsinstate,local,andprivatelabs;differencesinlaboratorycapacity(andgenerallylowerpublichealthcapacitiesinruralcounties)andthenatureoftests(PCRvs.viralculture);androutineprotocolsdidnotworkathighlevelsofthroughput/surge.Theexistenceofmultipledatasystems,ontheotherhand,mightbechangedtoprepareforfutureevents,orexpectedandcompensatedfor.

Takingthisintoaccount,theanalysisofpublichealthsurveillanceandlaboratorycapacitiessuggestedtwolessonsforimprovingpublichealthsysteminpreparationforfutureevents:theneedfor(1)clear,comprehensive,uniformdatasystemsand(2)a


29

centralincidentcommandsystemineachcountythatthencoordinateswiththestateregionalofficeandwiththeothercounties.

Mitigation

OncethescaleoftheWNVoutbreakwasknown,officialshadtomakedecisionsabouthowtocontrolthemosquitopopulation(objective).Inparticular,theresponsechallengewaswhethertoutilizeaerialspraying,whichhadnotbeendonehistoricallyintheDallas‐FortWorthMetroplex,ortousetruckspraying.Thisdecisionhadtobemadeseparatelyineachcountyandinconsiderationoftheindependenceofeachcountyandthecitiesineachcounty.

Fourimmediatecausesinfluencedthispublichealthsystem’sresponse.First,therewaspublicoppositiontoallformsofsprayingandespeciallytoaerialspraying.Second,therewerequestionsaboutthefinancialimplicationsofsprayingandonlylimitedexpertizeabouthowthisshouldbedone.Third,therewasonlylimitedmosquitotrackingdatatoguidedecisionsonwhereandwhentospray.Finally,thepressingtimeframerequireddecisionstobemade“onthefly.”


30

Avarietyoffactorscontributedtothissituation.Someareunchangeableatleastinthenearterm:publicconcernsaboutpesticidesandchemicalsingeneral,limitedpublichealthcapacitiesinmoreruralcountiessuchasDentonandTarrant,andnaturalhumanresistancetoaskingforhelpfromanorganizationoutsideofaperson’snormalrealm.Inaddition,althoughtherewasaneedtoactquicklybecauseofthetimescaleoftheoutbreak,therewerenopre‐existingdecisionmakingtriggers(i.e.ifthereareacertainnumberofidentifiedcasesthenaerialsprayingshouldtakeplace,etc.).Asaresult,therewasnotenoughtimetobuildconsensusamongdecentralizedcommunitymembers,organizationalstakeholderssuchasbusinessesorthehospital,andthevariouscounties.

Takingthisintoaccount,theanalysisofmitigationeffortssuggestedtwolessonsforimprovingpublichealthsysteminpreparationforfutureevents:theneedfor(1)clear,comprehensive,uniformdatasystemsthatincludemosquitodataand(2)science‐basedtriggersforspraying,aswellasmutualassistanceplanswithsprayingcompaniesandamongjurisdictionsfortimeswhensprayingisnecessary.Therootcauseanalysisdiagramforthiscapabilitycanbeviewedbelow.


31

Communicatingwiththepublic

ThefinalsetofobjectivesweretocommunicatewithresidentsabouthowtheycouldprotectthemselvesfromWestNilevirusandtocommunicatewithinthepublichealthsystemtocoordinateeffortsbetweenandamongstateandcountypublichealthandotherpublicandprivateorganizations.Theseobjectivessetuptworesponsechallenges.First,healthofficialsneededexternalsupportforunfamiliarmitigationefforts,whichrequiredaunityofpurpose.Second,althoughcomplicateddecisionswerebeingmadeindependentlyinthreecounties(e.g.DallasCountydecidedtoengageinaerialsprayingsignificantlybeforetheothertwocounties),therewasaneedtospeaktothepublicwithoneunifiedvoice.

Threeimmediatecausescontributedtotheseresponsechallenges.First,therewasconfusionaboutthedistinctionbetweenadisasterdeclarationandadeclarationofapublichealthemergency.Second,eachofthejurisdictionswascommunicatingseparatelywiththestatehealthdepartmentandCDCofficials.Finally,asnotedabove,wastheneedtomakediseasemitigationdecisionsonthefly.Toaddresstheseissues,electedofficialsandpublichealthpersonnelbeganholdingverybriefmeetingsbetweendecisionmakersandthosewithinformationondataorprocesses.Thesemeetings(adaptationsandsolutions)occurredbothwithinandbetweencountiesandhelpedtospreadinformationamongkeypersonnelasrapidlyaspossible.

Avarietyoffactorscontributedtothissituation.Onefactorwasunchangeableatthelocallevelatleastinthenearterm:thelimitedpublichealthcapacitiesinruralcountiesthataredependentonthestatehealthdepartment,whichcanalsobeoverwhelmed.Twootherfactorsreflectedalackofawareness,ofthedifferenceinstandardsandproceduresfordifferentkindsofdeclarations,andthatanemergencydeclarationandadecisiontouseaerialsprayingwerenotequivalent.Thefinalfactorwaspoliticalobstaclestoopencommunicationabouttheemergencydeclarationandaerialspraydecisionbecauseelectedofficialstaketheleadinmakingtheseofficialdecisions.

Takingthisintoaccount,theanalysisofmitigationeffortssuggestedthreelessonsforimprovingpublichealthsysteminpreparationforfutureevents:theneedfor(1)acentralincidentcommandsystemineachcountythatthencoordinateswiththestateregionalofficeand(2)mutualassistanceplanswithsprayingcompaniesandamongjurisdictionsfortimeswhensprayingisnecessary.Thisanalysisalsoidentifiedtheneedfor(3)ameanstoshareinformationwithotherjurisdictionslocally,withthestate,andnationally.


32


33

TheEvidenceBaseforPeerAssessmentinPublicHealthEmergencyPreparedness

Background

TheInstituteofMedicinehasdefinedthepublichealthsystemasthe“complexnetworkofindividualsandorganizationsthathavethepotentialtoplaycriticalrolesincreatingtheconditionsforhealth”(IOM2003).Forpublichealthemergencypreparedness(PHEP)system,theseorganizationsincludenotonlyfederal,stateandlocalhealthdepartments,butalsohospitalsandhealthcareproviders,firedepartments,schools,themedia,andmanyotherpublicandprivateorganizations(IOM2008).Inthelastdecade,organizationsinvolvedinpublichealthemergencypreparednesshaveworkedhardtoinnovateandimprovetheirprocesses,butmosthavenotoftensystematicallyanalyzedwhyinnovationswork(ordonotwork)orhaveaframeworkfordisseminatinglessonslearnedfromtheirexperience.Asaresult,“lessonslearned”frompublichealthemergenciesoftenremainunlearned—oratleastuntranslated—tonewemergenciesandneworganizations.Theconsequencesmaybeavoidablemorbidityanddeathsor,atbest,inefficientuseofresourcesinatimewhenpublichealthbudgetsaresmallandoftenshrinking.

Toaddresstheseproblems,andultimatelyimprovethepublichealthsystem’sabilitytorespondeffectivelytoemergencies,thefederalgovernment’sNationalHealthSecurityStrategy(NHSS)callsonthenation’spublichealthsystem—definedbroadlyaspertheIOM—toadaptsystematicqualityimprovement(QI)methodsandacultureofQItolearnfromexperienceinordertoenhancethehealthsecurityofournation(DHHS2009).Asinhealthcaremorebroadlyandotherindustries,effectivequalityimprovementinthePHEPsystemrequiresuseofrigorousanalyticalmethodsthatallowthesystem’sperformancetobeassessedandcomparedovertimeandbetweenjurisdictions.However,standardqualityimprovementmethodssuchaslearningcollaborativesthatarewidelyusedinhealthcaresettingsmaynotbeappropriateinthecontextofpublichealthemergencypreparednessduetothelackofestablishedevidence‐basedandagreeduponoutcomeandperformancemeasures,thedifficultyofcarryingoutrapidplan‐do‐study‐act(PDSA)cyclesandmeasuringassociatedoutcomes(Stoto,2013a).

Partoftheproblemisthatpublichealthemergenciesarerelativelyrareandgenerallynotrepeatedinthesamemannerandcontext.Whileroutinehospitalservicescanbestudiedandimprovedwithstatisticalprocessandoutcomemeasures,systemimprovementforrareeventsrequiresthein‐depthstudyofindividualcases(Berwick,2003).Whenthefocusisonimprovementratherthanaccountability,andoncomplex

Appendix A Evidence Base for Peer Assessment


34

PHEPsystemsratherthantheircomponentsorindividuals,qualitativeassessmentofthesystemcapabilitiesofPHEPsystemscanbemoreusefulthanquantitativemetrics(Stoto,2013a).Qualitativemethodscanhelpprobehowandwhythingshappenbyhelpingilluminatehowcausalmechanismsaretriggeredinvaryingcontexts.

EffectivePHEPsystemimprovement,similarly,requiressystematicmethodsforlearningfromindividualorganizations’experienceinunusualsituations.Thechallengesoflearningfromexperienceisnottodeterminewhatshouldhavebeendone,butrathertoidentifyandaddressunderlyingfactorsthatcouldlimitthesystem’sabilitytorespondeffectivelytofutureevents,whicharelikelytobedifferent.Forinstance,whilethespecificchallengesexperiencedduringthe2009H1N1influenzapandemicwerenew,theresponserequiredcombinationsofsomeofthebasiccapabilitiesthathadbeentestedinprioremergencies,suchastheneedtomaintaincommunicationswiththepublicorprovideepidemiologicsituationalawareness.

Thelimitationsofcurrentapproachesalsoreflecttheintrinsicdifficultyoflearningfromuniqueevent.Oneproblem,forinstance,isthelackof“counterfactuals,”knowledgeofwhatwouldhavehappenedunderanotherresponse.Analysesofpublichealthemergencyresponses,moreover,musttakeintoaccountadditionalchallengessuchasthecommonneedforamulti‐jurisdictional,multi‐sectoralresponse,involvingtheentireemergencypreparednesssystem,makingitisdifficulttoknowwhatthebestapproachwouldhavebeen(Stoto,2012).Heretoo,qualitativemethodscanbeausefulcomplementtoquantitativeapproaches,whosestrengthliesinidentifyingpatternsofvariationin,andcovariationamong,variables.

Althoughqualitativemethodsareoftencriticizedasinsufficientlyrigorousandtransparent,thereisawell‐establishedbodyofsocialsciencemethodsthatcanhelptoensurerigorinqualitativeresearch.Gilsonandcolleagues(2011)summarizeaseriesofconcreteprocessesforensuringrigorincasestudyandqualitativedatacollectionandanalysis(seeBox1).Becausethefocusisonpublichealthsystemsratherthanindividuals,Yin’s(2009)classicbookoncasestudymethods,nowinits5thedition,isalsorelevant.Marchandcolleagues(1991)andWeickandSutcliffe(2001)offermorespecificsuggestionsforstudyinguniqueincidents.Stoto,Nelson,andKlaiman(2013)discusstheapplicationofthesemethodstoPHEP.Theystressafamilyof“theoryoriented”evaluationmethodsthatuseprogramtheorytoguidequestionsanddatagathering,andwhichinsistonexplicatingthetheoriesormodelsthatunderlieprograms,elaboratingcausalchainsandmechanisms,andconceptualizingthesocialprocessesimplicatedintheprogram’outcomes(Dixon‐Woods,2011).Onewell‐knownmemberofthisfamilyisknown“realistevaluation.”DevelopedbyPawsonandTilley(1997),thisperspectiveplacesthefocusofresearchandevaluationlessonrelationshipsamongvariablesthanamongexplorationofthecausalmechanismsthatgenerateoutcomes.Theperspectivealsorecognizesthatmechanismsthatcanbe“fired”successfullyinonecontextmaynotworkwellinothercontexts.

Insupportofthepeerassessmentapproachpresentedinthistoolkit,thissectionpresentsfourspecificapproachesthatdrawonthissocialscience/evaluationtheory


35

literature:casestudyresearch,peerassessment,rootcauseanalysis,andfacilitatedlookbacks.ThefollowingsectiondiscusseshowanalysesbasedontheseapproachescanbesharedandfurtherstudiedinaCriticalIncidentRegistry.

Casestudyresearch

Stotoandcolleagues(2005)usedaqualitativecasestudyapproachofthepublichealthresponsestoWestNilevirus,SARS,monkeypox,andHepatitisAtomeasurePHEPgoalsandobjectives,aswellasthecapabilitiesandcapacity‐buildingactivitiesintendedtoassurethosegoalsandobjectives.Theyconductedsitevisitsinsixstates,andatleasttwolocalareaswithinthosestates,thatillustratedtherangeofthepublichealthresponse.Site

Box 1. Processes for ensuring rigor in case study and qualitative data collection and analysis (adapted from Gilson, 2011) Prolonged engagement with the subject of inquiry. Although ethnographers

may spend years in the field, HPSR tends to draw on lengthy and perhaps repeated interviews with respondents, and/or days and weeks of engagement within a case study site.

Use of theory. To guide sample selection, data collection and analysis, and to draw into interpretive analysis.

Case selection. Purposive selection to allow prior theory and initial assumptions to be tested or to examine ‘‘average’’ or unusual experience.

Sampling. Of people, places, times, etc., initially, to include as many as possible of the factors that might influence the behavior of those people central to the topic of focus (subsequently extend in the light of early findings) Gather views from wide range of perspectives and respondents rather than letting one viewpoint dominate.

Multiple methods. For each case study site: Two sets of formal interviews with all sampled staff, Researcher observation & informal discussion, Interviews with patients, Interviews with facility supervisors and area managers.

Triangulation. Looking for patterns of convergence and divergence by comparing results across multiple sources of evidence (e.g., across interviewees, and between interview and other data), between researchers, across methodological approaches, with theory.

Negative case analysis. Looking for evidence that contradicts your explanations and theory, and refining them in response to this evidence.

Peer debriefing and support. Review of findings and reports by other researchers.

Respondent validation. Review of findings and reports by respondents Clear report of methods of data collection and analysis (audit trail). Keeping a

full record of activities that can be opened to others and presenting a full account of how methods evolved to the research audience.


36

visitswereconductedbytwo‐orthree‐memberteamsofRANDscientistsusingadetaileddiscussionguideasdescribedbelow.Theseteamscollectedinformationthroughin‐personinterviewswithseniorofficialsoftheselectedhealthdepartmentsandothersinthecommunitiesnominatedbythehealthdepartments.Theseinterviewswereconductedindividuallyoringroups,dependingonthepreferencesofthestateandlocalofficials.Telephoneinterviewswerescheduledwithkeyinformantswhowerenotavailableduringthevisit.Before,during,andafterthesitevisits,teammembersgatheredinformationregardingthenatureandextentofthepublichealthdepartments’emergencyresponseactivitiesforasmanyoftheoutbreaksaswererelevant,includinginformationonthedepartments’interactionswithHHSandotherfederalagenciesandotherpublicorprivateentities.Theteamsalsogatheredmaterialspreparedbythehealthdepartmentsduringtheoutbreaks(forbothinternalandexternaluse),analysesextractedfromrelevantreportsandpublications,andothermaterials.

Peerassessment

Ensuringobjective,systematic,andreliableanalysesofcriticalincidentscanbechallengingifhealthofficialsareevaluatingtheirownresponse(Piltch‐Loeb,2013).Asanalternative,evaluationbypeersinsimilarjurisdictionsoffersthepotentialforobjectiveanalysesbyprofessionalsfamiliarwithpublichealthpreparednessaswellastheparticularitiesofthesystem’sbeingassessedand,atthesametime,canbeaneffectivewaytosharebestpracticestosupportandamplifytechnicalassistanceprovidedbyCDC.

Ourresearchhasindicatedthatsuchpeerassessmentscanbebothreliableandobjective.Inoneexample,theHealthOfficersAssociationofCalifornia(HOAC,2007)conductedin‐depthemergencypreparednessassessmentsin51ofthestate’s61LHDs.LHDsincludedinthisreportwereassessedbetweenNovember8,2005andOctober26,2006.TheprimarypurposeoftheprojectwastoassesspublichealthemergencypreparednessineachLHDrelativetospecificfederalandstatefundingguidanceandidentifyareasneedingimprovement.Astructuredassessmentinstrument,keyedtotheCDCandHRSA2005‐06Guidance,wasusedtoexaminetheextentofLHDcapacityandprogressinpreparedness.Theinstrumentincludedperformanceindicatorsanda4‐pointscoringrubric(fromminimallytowell‐prepared)forquantifyingtheresults.Teamsofthreetofourconsultantsfromasmallcorpsofexpertpublichealthprofessionalsrecruitedforthisprojectmade2‐daysitevisitstotheLHDsthatvolunteeredtoparticipateintheassessment.TheassessmentmethodsincludedutilizingtheassessmenttooltoguideinterviewswithmultiplelevelsofLHDstaff,reviewinglocalpreparedness‐relateddocuments,anddirectlyobserving.AnLHD‐specificwrittenreportoffindingsandrecommendationswaspreparedandsenttoeachparticipatingLHDwithin6‐8weeksofthesitevisit.

RootCauseAnalysis

Rootcauseanalysiscanbeusedtosupportmanyofthenewapproachesalreadydiscussed.Manystrategieshavebeendescribedfordeep,probinganalysesaboutwhatcausedanegativeoutcomeorengenderedapositiveone.Rootcauseanalysis(RCA)is


37

familiartomanyinthehealthcaresectorbecauseboththeJointCommissionandDepartmentofVeteransAffairsrequireRCAsforcertainclinicalevents(Wu,2008).ThegeneralgoalofRCAsistomovefromsuperficial,proximatecausestosystem‐levelrootcausesbyrepeatedlyaskingwhyeachidentifiedcauseoccurred(Crouteau,2010).Inprinciple,RCAsshouldfacilitatesignificantlybetterlearningfromasingleincident,buttheysometimesfailtodoso.Inparticular,researchershaveidentifieddiscrepanciesinanalysis,dependentonexperienceandorganizationalexpectations;limitedrigor;andpracticalbarriers,suchasscarcityoftimeandresources(Wu,2008).Thereisoftenatendencytosimplifyexplanationsaboutcriticalincidents,eitherbydiscountinginformationthatdoesnotconformtopre‐existingbeliefsorbyfailingtoexamineaproblemfrommultipleperspectives(Weick,2001)andtoblamefailuresonsituationalfactorsinsteadofidentifyingopportunitiesforsystemsimprovement(Edmondson,2010).Whilethesearenotinevitableproblems,theydohighlighttheneedfortoolsandprocessesintendedtohelprootcausestoberesponsivetocommonchallengesfacedwhenconductingRCAs,andhighlightthepersistentissuesthatarisefromalackoftraininginretrospectiveanalysis.

Toillustratetheapproachtorootcauseanalysisdescribedinthistoolkit,considerthecasestudythatourgrouppreparedonthepublichealthsystemresponsetothe2009H1N1pandemiconMartha’sVineyard,anislandof16,000residentsinMassachusetts(Higdon,2013).Thepublichealthemergencyresponseobjectivewastovaccinatethepublicinanefficientandorganizedmannersuchthathighprioritygroupsarevaccinatedfirstandtherestofthepubliciscoveredasmuchaspossible,anexampleofCDCPHPCapability:MedicalCounter‐measuredispensing.AsisdiscussedinmoredetailinHigdonandStoto(2013),thesixtownhealthdepartmentsregisteredasasingleentitytoreceivevaccinesuppliesfromthestate,butseparatelyfromtheisland’sonlyhospital,onepediatrician,andtheWampanoagIndiantribe.AninformalMartha’sVineyardpublichealthcoalition(MVPHC),representingthetownsandthetribebutnotthehospitalorotherproviders,plannedasingleisland‐widevaccineclinic.Butwhenvaccinedeliveriesweredelayedandcameinsmallunits,thecoalitionswitchedtoaschool‐basedvaccinationstrategyinwhich“shooter‐teams”drawingonallofthetownsvaccinatedthechildrenineachschooldependingonthenumberandtypeofvaccinethatarrived.Thehospitalwasnotawareofthechangeinplansandmadeinconsistentassumptionsaboutwhichorganizationwouldprovidevaccineforvulnerablepopulationotherthanschoolchildren,whichcausedconfusion.Aroot‐causeanalysis,summarizedinthediagramonthenextpage,suggestedthefollowinglessonsfromthisexperience.

• Localimplementationofthevaccinationcampaignshouldbeflexible,allowingsharingpersonnelandresourcesacrosstownstoconstituteshooterteams,

• Stateofficialsshouldrecognizeregionalcoalitionspriortoanemergency• HospitalandotherprovidersshouldbemorefullyinvolvedwiththeMVPHC.

FacilitatedLookbacks

Afacilitatedlookbackisanestablishedmethodforexaminingpublichealthsystems’emergencyresponsecapabilitiesandforconductingacandidsystems‐levelanalysis.


38

Throughtheuseofaneutralfacilitatorandano‐faultapproach,dimensionsofdecisionsareprobedandnuancesinpastdecision‐makingexploredindetailthroughdiscussionswithpublichealthleadersandkeystaff,aswellasavarietyofcommunitystakeholdersinvolvedwiththeresponse.Abriefchronologyoftheeventsthatoccurredisreviewedduringthissession.Thefacilitatorguidesthediscussionandasksprobingquestionssurroundingkeyissuesaboutwhathappenedatvariouspointsinthechronologythatwaspresented,keydecisionsthatweremadebyvariousstakeholders,andhowdecisionswereperceivedandacteduponbyothers,andlessons‐learnedareelicited(Aledort,2006).

Bywayofillustration,wedescribeafacilitatedlookbackmeetingthatwasconductedinBostoninMay,2010,oneofthreesuchmeetingsconductedtoprovideinformationfortheMassachusettsDepartmentofPublicHealth2009H1N1AAR/IP.Theobjectiveswere(1)toconductsystems‐levelanalysisofstate/local/privatesectorresponsesto2009‐2010H1N1pandemic;(2)toidentifylessonsapplicabletobothpandemicandpublichealthpreparednessgenerally,and(3)toidentifyopportunitiesandgoalsforimprovement.Thegroundrulesstressedtheimportanceofa“systemsimprovementspirit,”specifically:

• Thereisnoperfectresponsetopandemic• Participantswillhavedifferentperspectives• Openandhonestdialogueandfeedback• Askquestionsandchallengeoneanother

Story Arc: Towns on Martha’s Vineyard register to receive flu vaccine independently of the Martha’s Vineyard Hospital and physician’s offices. An island wide clinic was planned for and then delayed twice due to delays in vaccine produc on and delivery. In response, shooter teams were organized to vaccinated children in schools

Objec ve: Vaccinate the public in an efficient and organized manner such that high priority groups are vaccinated first and the rest of the public is covered as much as possible Capability: Medical Counter‐measure dispensing

Lessons Learned: • Local

implementa on of the vaccina on campaign should be flexible, allowing sharing personnel and resources across towns to cons tute shooter teams,

• State officials should recognize regional coali ons prior to an emergency

• MVH and other providers should be more fully involved with the MVPHC.

Necessary informa on was not included in the original state‐wide vaccine registry system because the system was for children's vaccines

only.

Despite efforts to organize the town HDs, MVH and its associated clinics were not

communica ng well.

Vaccine registra on system was not organized to report by geographic units other than MDPH coali ons.

Manufacturing delays and mul ple vaccine

formula ons did not allow for vaccine to be delivered

as planned.

Vaccine shipments were mismatched by loca on and

adult versus child’s doses

Incorrect assump ons about which organiza on would provide vaccine for vulnerable

popula on other than school children

There was confusion about

who was responsible for vaccina ng

pregnant women and preschool

children.

Informal MV Public Health Coali on (MVPHC) adapts by developing a school‐based vaccina on approach in which Island‐wide “shooter teams” travel from school to school to administer vaccines.

Vaccine arrived in small batches rather than a

single large batch.

Directly impacted whether the objec ve was met

Shaped the lessons learned and resul ng improvement plan

Response challenges Immediate causes Contribu ng factors

Immediate reason(s) for response challenges that affected mee ng objec ve

Underlying factors (modifiable & un‐modifiable) that lead to the immediate cause


39

• Noonesingledout,blamed,orpenalized• Actionplanbasedonwhatislearnedisneeded.

Followingareviewofthetimelineoftheresponseandsomestatisticaldata,

participantsaddressedfivesetsofissues:vaccineprioritiesanddistribution;vaccineadministrationandclinicmanagement;staffingandsurgeissues;communicationwiththepublic;andcommunicationwithinthepublichealthsystem.Foreachofthesetopics,thefollowingcorequestionsframedthediscussion:

• Briefoverviewofyour“normal”activities,rolesandresponsibilities• Whatwentwell?Why?• Whatcouldhavegonebetter?Whatcouldhavebeendonedifferently?Whatwere

theunderlyingproblems?• Whatsystemschangesareneededtoimprovefutureperformance?

Togetherwithinformationfromaseriesofindividualandgroupinterviews,aswell

astheMartha’sVineyardcaseabove(HigdonandStoto,2013),informationfromthefacilitatedlookbacksledtotheidentificationofbothsuccessesandproblemsintheMassachusettsresponseefforts.Mostofthechallengescamefromtheinabilityofhealthofficialstotrackandforecastvaccinedistributioneffectively.Largehealthcaredeliveryorganizationsmadeuseofelectronicmedicalrecordstoautomaticallyschedulevaccinationappointmentsandnotifypatients.Communityhealthcenterswithlessadvancedtechnologytookadvantageoftheirrelationshipwiththeirclientstoovercomereluctancetoacceptthevaccine.Despiteafracturedpublichealth“system”onMartha’sVineyard,localhealthandschoolofficialsworkedtogethertoshareresourcessuchasvaccinationteamsthatwouldgofromoneschooltoanotherasvaccinebecameavailable.Theanalysisalsoillustratesthechallengesofmanagingthedistributionofvaccinewhenthetimingandamountofvaccinetobedeliveredwasuncertainandidentifiesanumberoflessonsaboutcommunityresilience.Inparticular,itshowstheneedtobalanceprecisepolicieswithflexibleimplementationaswellastheimportanceoflocalinvolvementindecision‐makingandincreasingthetransparencyofcommunications.Thecaseillustratestheimportanceofbuildingcommunitycapacitystrongcommunity‐widepartnershipstoaddresspersistentpublichealthproblems.Moregenerally,thecaseillustratesboththevalueofpreviousinvestmentsinbuildingthepublichealth’ssystem’ssocialcapitalaswellastheneedformore.ACriticalIncidentRegistryforPHEP

Otherfieldshavefoundwaystolearnfromrareevents,andtheymayprovideamodelforthepublichealthemergencypreparednesssystem.Aviationisprobablythemostprominentexample.Inthemid‐1970s,almost1,000peoplediedayearfromaircrashesaroundtheworld.Today,thatnumberhasbeencutinhalfeventhoughthenumberofflightshasdramaticallyincreased(Boeing2012).Airsafetyhasimprovedinlargepartbecauseoftheuseofcriticalincidentregistries(CIRs),whichareusedtoidentifyandsystematicallyanalyzerareevents—andresponsestothem—todrivelearningandsystemsimprovement.Throughuseofsuchregistries,theairlineindustryhasbecomeadeptatdrawingsystem‐widelessonsfromsingleincidents,andpiecingtogethersystem


40

improvementsfromseeminglyinnocuousoccurrencesobservedacrossmultipleaccidentsorclosecalls(Wald2007).

Becauseoftheirsuccessinaviationsafety,criticalincidentregistrieshavealsobeen

adoptedinotherindustries,includingothersectorsoftransportation,healthcare,andworkplacesafety.TheyarecurrentlyusedbypublicandprivatesectororganizationsincludingtheFederalAviationAdministration,NationalTransportationSafetyBoard,NationalAeronauticsandSpaceAdministration,theFoodandDrugAdministration,andtheFederalBureauofInvestigation.InhealthcaretheJointCommission,theVeteransHealthAdministrationandtheNewYorkChapteroftheAmericanCollegeofPhysiciansmaintainregistries.Thoughtheseregistriestakedifferentforms,dependingonthepracticalcontext,allaredesignedtofacilitatelearningfromrelativelyinfrequentevents.ThemajorapplicationsofCIRsindifferentsectorsinclude:(1)understandingcontextsandmechanismsthatdrivesuccessfulandunsuccessfulpracticeswithinsystem;(2)identifyingandsharingbestpractices;(3)drivingindividualandorganizationalimprovement;and(4)describingthefrequencyandnatureofincidents.Animportantpartofthesuccessofcriticalincidentregistriesistheiruseofrootcauseanalysis,whichmovesfromsuperficial,proximatecausestosystem‐levelrootcausesbyrepeatedlyaskingwhyeachidentifiedcauseoccurred(Croteau2010).

Thesuccessofcriticalincidentregistriesinotherfieldssuggeststhataproperly

designedPHEPCIRcouldsupportbroaderanalysisofcriticalpublichealthincidents,facilitatedeeperanalysisofparticularincidentsandstrongerimprovementplans,andhelptosupportacultureofsystemsimprovement.Inparticular,byencouragingrootcauseanalysesandsharingtheresultsofthoseanalyseswithothersthroughadatabase,aPHEPCIRcouldbeavaluableapproachforsystemsimprovement.Itmayalsofacilitatebetterinvestmentofscarceresourcesinapproachesmostlikelytobeeffective.

However,whilesystemsforsharinglessonsfromemergencyresponsescurrently

exist,nonehaveallofthecharacteristicsofsuccessfulCIRs.Forinstance,inordertosharelessonslearnedaboutpublichealthpreparedness,theNationalAssociationofCountyandCityHealthOfficialsmaintainsacollectionoflocalhealthdepartments’reportsof“successfulpractices”fromtheH1N1pandemicandotherevents.Thiscollectionisdesignedforquicksuggestionsandhypothesisgenerationaboutbestpractices,butrootcauseanalysisandin‐depthevaluationsareneitherrequirednortypicallyincluded.TheDHSLLISsystemincludesrelativelyfewpublichealthincidentsand,asillustratedwiththeproblemsobservedbySavoiaandcolleagues(2012)intheiranalysisdoesnotencouragedeepanalysis.Indeed,aswasseeninthe2009H1N1AAR/IPsdiscussedabove,rootcausesareoftennotexploredinPHEP.Thus,thereremainsaneedforanapproachthat,inadditiontocapturingeventsandresponsesinamoreusableway,alsoprovidesananalyticalmethodforextractinglessonsfromreportsabouthowthePHEPsystemrespondedtospecificincidents.

Criticalincidentregistriesconsistofadatabaseofreportsabouttheresponsetoindividualincidentsthataresubmittedbythepublichealthagenciesrespondingtotheemergency.AsillustratedinFigure1,peerassessmentsareonewaythatincidentreportsmightbegenerated.


41

Tobeeffective,registriesrequirestandardprotocolsforanalysisofcriticalevents

andprobinganalysisthatcanenablethetranslationoflessonstoidentifypatternsofsuccessesandfailuresareneeded.Inordertoprovideenoughinformationforacredibleanalysis,experienceinotherfieldsanddiscussionswithpublichealthpractitionerssuggestthatstandardCIRreportsshouldconsistoffourparts:abriefsummary,abackground/contextsection,adescriptionoftheincident,andananalysisofthePHEPsystem’sroleintheincident.Theabstract‐lengthsummaryiscondensedoverviewoftheincident,whatPHEPcapabilitiesweretested,significantcontextualfactors,andthekeyfindingsderivedfromanalyzingtheincident.Itwouldbesearchablebyresearchersandpractitionersseekingtoidentifytrendsacrosssimilarincidentsorlessonsthatmightbeapplicabletoacurrentoranticipatedincident,asiscommoninthethreeaviationdatabases.Thebackgroundwouldprovidethecontextualinformation(suchasthetypeandsizeofthehealthdepartment,communityresourcesandcharacteristics,natureofthepathogen)neededtoanalyzethesystem’sperformance.Theincidentdescriptionprovidesaconciseexplanationoftheincident,includingatimelineofhowtheincidentunfoldedandrelevantchangesthroughouttheincident,andsufficientlydetailedonboththeincident’skeyeventsandhealthsystemcontextinordertocriticallyanalyzethePHEPsystem’sresponse.Finally,theanalysissectionemploysroot‐causeanalysistoprovideadeepanalysisofwhetherandwhyparticularPHEPcapabilitiesweresuccessfullyemployedinresponsetotheincident(Piltch‐Loeb,2013).UnliketypicalAAR/IPs,CIRreportsdonotneedtoincludeeverycorrectiveactionidentified.Rather,sincethepurposeistoidentifyissuesandpotentialsolutionsthatmightberelevantinothersituations,solutionsthatseeminnovativeandpotentiallygeneralizabletosimilarjurisdictionsshouldbeincluded.


42

REFERENCES AledortJE,LurieN,RicciK,etal.(2006).Facilitatedlookbacks:Anewqualityimprovement

toolformanagementofroutineannualandpandemicinfluenza,RANDTR‐320.Availableat:http://www.rand.org/pubs/technical_reports/TR320.html

BarnettDJ,BalicerRD,ThompsonCB,etal.(2009).Assessmentoflocalpublichealthworkers'willingnesstorespondtopandemicinfluenzathroughapplicationoftheextendedparallelprocessmodel.PlosONE4(7):e6365.

BohmerRJ(2010).Fixinghealthcareonthefrontlines.HarvardBusinessReviewApril:62‐69.

BrownT(2008).Designthinking.HarvardBusinessReviewJune:84‐92.CentersforDiseaseControlandPrevention(CDC,2011).PublicHealthPreparedness

Capabilities:NationalStandardsforStateandLocalPlanning.Availableat:http://www.cdc.gov/phpr/capabilities

CroteauRJ(2010).RootCauseAnalysisinHealthCare:ToolsandTechniques.JointCommissiononAccreditationofHealthcareOrganizations.

DepartmentofHealthandHumanServices(DHHS,2009).NationalHealthSecurityStrategyoftheUnitedStatesofAmerica.Availableat:http://www.phe.gov/Preparedness/planning/authority/nhss/Pages/default.aspx

GilsonL,etal.(2011).Buildingthefieldofhealthpolicyandsystemsresearch:socialsciencematters.PLoSMedicine8:e1001079.

HigdonMA,StotoMA(2013).TheMartha’sVineyardpublichealthsystemrespondsto2009H1N1.InternationalPublicHealthJournal5(3):369‐378.

HealthOfficersAssociationOfCalifornia(HOAC2007).EmergencyPreparednessInCalifornia’sLocalHealthDepartments.HealthSanFrancisco(pp.1‐23).Availableat:http://www.usc.edu/schools/price/bedrosian/private/docs/ccg_papers/callahan.pdf

InstituteofMedicine(2008).ResearchPrioritiesinEmergencyPreparednessandResponseforPublicHealthSystems(letterreport).Washington,DC:NationalAcademiesPress.

KlaimanT,KraemerJD,StotoMA(2011).Variabilityinschoolclosuredecisionsinresponseto2009H1N1.BMCPublicHealth11:73.

KlaimanT,O’ConnellK,StotoMA(2013).Localhealthdepartmentpublicvaccinationclinicsuccessesduring2009pH1N1.JournalofPublicHealthManagementandPractice.19:E20–E26.

LurieN,WassermanJ,StotoMA,etal.(2004).Localvariationinpublichealthpreparedness:LessonsfromCalifornia.HealthAffairsOnlineW4‐341‐353.

MarchJG,SproullLS,andTamuzM(1991).OrganizationScience2:1‐13.NationalAssociationofCityandCountyHealthOfficials(NACCHO,Updated2009).Case

studiesoftheMRCprogramcollaborationwithlocalhealthdepartments.Availableat:http://www.naccho.org/topics/emergency/MRC/resources/upload/Burness‐report_final031809.pdf.

NationalHealthSecurityStrategy(NHSS,2009).Availableat:http://www.phe.gov/Preparedness/planning/authority/nhss/strategy/Documents/nhss‐final.pdf

PawsonRandTilleyN(1997).RealistEvaluation.London:Sage.


43

Piltch‐LoebR,KraemerJD,StotoMA(2013).Synopsisofapublichealthemergencypreparednesscriticalincidentregistry(CIR),JournalofPublicHealthManagementandPractice,2013;19(5),S93–S94.

SavoiaE,AgboolaF,BiddingerPD(2012).Useofafteractionreports(AARs)topromoteorganizationalandsystemslearninginemergencypreparedness.InternationalJournalofEnvironmentalResearchandPublicHealth,9:2949‐2963.

StotoMA,DauseyD,DavisL,etal.(2005).Learningfromexperience:ThepublichealthresponsetoWestNileVirus,SARS,Monkeypox,andhepatitisAoutbreaksintheUnitedStates.RANDTR‐285.Availableat:http://www.bvsde.paho.org/bvsacd/cd57/RAND_TR285.pdf.

StotoMA,NelsonCD,andtheLAMPSinvestigators(2012).MeasuringandAssessingPublicHealthEmergencyPreparedness:AMethodologicalPrimer,September,2012.Availableat:http://lamps.sph.harvard.edu/images/stories/MeasurementWhitePaper.pdf.

StotoMA,CoxH,HigdonMA,DunnellK,GoldmannD.(2013a).UsingLearningCollaborativestoImprovePublicHealthEmergencyPreparednessSystems.FrontiersinPublicHealthSystemsandServicesResearch,Vol.2:No.2,Article3.

StotoMA,NelsonC,HigdonMA,KraemerJD,SingletonCM.(2013b).LearningAboutAfterActionReportingfromthe2009H1N1Pandemic:AWorkshopSummary.JournalofPublicHealthManagementandPractice,17:1‐7.

StotoMA,NelsonC,HigdonMA,etal.(2013c).LessonsabouttheStateandLocalPublicHealthSystemResponsetothe2009H1N1Pandemic:AWorkshopSummary.JournalofPublicHealthManagementandPractice17:1‐7.

StotoMA,NelsonC,KlaimanT(2013d).Gettingfromwhattowhy:Usingqualitativemethodsinpublichealthsystemsresearch.AcademyHealthPHSRIGissuesbrief.Availableat:http://www.academyhealth.org/files/publications/QMforPH.pdf

WuA,LipshutzAKM,PronovostPJ(2008).Effectivenessandefficacyofrootcauseanalysisofmedicine.JAMA,299:685‐686.

WeickKandSutcliffeK(2001).ManagingtheUnexpected:AssuringHighPerformanceinanAgeofComplexity,Jossey‐Bass.

YinR(2009).CaseStudyResearch:DesignandMethods,Sage.


44

PreparingforapeerassessmentAssoonaspossiblefollowinganincident: Preparinganinitialreport.Thereportshoulddescribetheincidentyouwouldlike

assistancereviewing,includingthetypeofincident,capabilitiesassessed,andchallengesinvolved.Insomecases,adraftAARthatcanservethispurposemayalreadyhavebeencompleted. ThereportshouldroughlyfollowtheCIRentryoutline:abriefsummaryofthe

incidentincludingPHEPcapabilitiestestedintheincident;abackgroundsectionthatincludestherespondingdepartment’sexperiencewiththistypeofincident;adescriptionoftheincidentthatincludesinvolvedparties;andaninternalassessmentoftheresponsetotheincident.

Withinsixmonthsoftheincident: Requestinganassessment.Youshouldcontactaselectgroupofpeersthatyouthink

wouldbeappropriatepeerassessmentteammembers.Threepeerassessmentteammemberswillbeyourtargetnumberfortheassessment.(IfaCIRexistsandhasadministrativestaff,youshouldworkwiththeseadministratorstoidentifyappropriatepeersandinitiatediscussions). Considerindividualswhoarefromsimilarjurisdictionsorhaveexperiencedsimilar

incidentsthemselves

Approximately1Monthbeforethesitevisit: Schedulingsitevisits.Youandtheassessmentteamswillcoordinateasitevisittime.Site

visitswillideallybeatleastonefulldayandincludethethreepersonpeerassessmentteam. Tofindatimethatworksforallparties:

o Youwillprovidepotentialdatesforsitevisitsandcommunicatewithpeerassessmentteamsviaemailorphone Thedatethatworksforallinvolvedwillbeselectedandyouwillnotify

therequestorsthatthisdateworkso Onceadateisselectedyouwillidentifyalocationforthemeetingtotake

place Shouldanovernightvisitbenecessary,youwillthenworkwith

requestorstofindahotelincloseproximitytothemeetinglocation Note:CIRstaffcanassistinhotelreservationsandtransportation

reservationswhenapplicable

Establishingameetingmethodology.Tomaximizethetimeandskillsetoftheassessmentteams,youwillmostlikelywanttoplanafacilitatedlookbackmeeting.

Appendix B Requestor Job Action Sheet


45

Whilethelookbackisnotrequired,ithasbeeneffectivelyusedatbothsitevisits.Youshouldplanaccordingly,andconsider: Invitingthosewhoplayedkeyrolesintheresponse,aswellasparticipantswith

varyingperspectives.Suggestedmeetingparticipantsinclude:o Leadersoftheemergencyresponsefromrespondingjurisdictionso Emergencyresponderso Representativesfromthelocaljurisdictionsoutsideofexplicitlypublichealth

activities(environmental,publicworks,etc.),dependingonthetypeofincident

o Statehealthworkersinvolvedintheresponseo Volunteercoordinatorso Hospitalrepresentativeso Homehealthrepresentativeso Coalitionleaderso Governmentworkersinvolvedinemergencydeclarationactivities(if

applicable) Creatinganinvitationthatincludes:thepurposeofthemeeting,thetimeand

locationofthemeeting,the“spiritofsystemsimprovement”beingexploredinthemeeting,andtheno‐faultzonethatwillexistduringthemeeting.Asampleinvitationisinthetoolkit

Approximately2weeksbeforethesitevisit: Documentreview.Priortothesitevisit,youwillsendtheassessmentteamsadditional

informationaboutyourjurisdictionandincident Additionaldocumentsthatmayberelevanttosendinclude:

o Relevantplans(tohelpassessmentteamsunderstandresponseactivitiesandidentifyopportunitiesforimprovement). Itemstoconsiderincludeincidentlogs,timelines,etc.

o Documentationofestablishedpartnershipsactivatedduringtheincident(toprovideadditionalcontext)

o Dataonrelevanthealthandsignificantnon‐healthoutcomes,responseactivities,etc.

o Mediareportso Timelineofmajoreventso Currentafteractionreports(shouldanyexist)o Otherrelevantdocumentsatthediscretionoftherequestingjurisdiction

One‐on‐OneDiscussionwithAssessmentteams.Priortothesitevisit,youwilldiscussyourunderstandingoftheincidentwiththeassessmentteams.Thisdiscussionwillservetoestablish:

o Themosteffectiveon‐sitemeetingformat Confirmifyouarecomfortablewiththefacilitatedlookbackprocess(see

toolkit) Doyoubelieveoneononeinterviewsarenecessaryaswell? Willtherebetimeforpre‐andpost‐debriefswiththeassessmentteams?

o Keypointsfordiscussionduringthesitevisit


46

Youshouldshareyourthoughtsontheincident Youshouldupdatetheassessmentteamsastowhatthepartieswhoare

attendingthemeetingareawareof Youshouldestablishwhatcapabilitieswerestressedduringtheincident

o Whatinformationthemeetingparticipantsareawareof HasthedraftAARbeensharedwiththeseparties? Isthereanysensitiveinformationnottobediscussed?

Approximately1weekbeforethesitevisit: Establishattendees.Createaroughattendancelistforthemeetingincludingthenames

ofparticipantsandtheirroles.Sendanemailremindertoallwhowereinitiallyinvited.Somereminderstoconsider:

o Areallkeypartiesrepresentedbasedonyourdocumentreview?o Howwillyouidentifyparticipantsatthemeeting?

Createtentnamecardsforallthoseexpectedtoattendandbringthesetothemeetingalongwithblanktentcardsforanylastminuteattendees

Establishmeetingschedule.Createaroughagendaforthepeerassessmentteamsvisitincludingthetimeframeforthefacilitatedlookbackandthetimeframetohaveindividualdiscussionsifyouandtheassessmentteamshavedeemedthatnecessary.Orderinglunchfromalocalcaterertobedeliveredtothemeetingsiteorbuildinginablockoftimeforparticipantstogettheirownlunchwillhelpthemeetingschedulerunsmoothly.

PeerAssessmentactivitieson‐siteApproximately1hourbeforethemeetingbegins: Day‐ofmeetingpreparation.ThemeetingspaceshouldincludePowerPointdisplay,

boardsthatcanbewrittenonduringthediscussion,andaroundtablesetup.Youwillideallyset‐uptheroominthefollowingway:o Chairsintheroomshouldbeorganizedsoparticipantscanfaceeachothero Tentcardsshouldbeon‐siteforparticipantswhoattendtowritetheirnameand

roleonandshouldthenbedisplayedateachindividual’sseat. Youshouldhaveatentcardaswell

Meetingbegins: MeetingDuration.Duringthemeetingyouareexpectedtoparticipateinthesame

fashionasotherparticipants.Considertheassessmentteamsquestionsthoughtfullyandengageasyouseefit.Yourtoolkitprovidesinformationontheslidesthataretobeusedduringthemeeting.

MeetingEndsafterapproximately5hours: PostMeetingOn‐sitefollowup.Ifpossible,afterthemeetingyoushouldmeetwiththe

assessmentteamtoshareyourperspectiveandclarifyanythingthatmaynothavebeenclearduringthemeeting.Itisatyourdiscretionwhattodiscussatthispoint

SiteVisitFollowUp


47

Withinapproximately2monthsaftersightvisit: Discussingfindings.Youwillbesentananalysisreportwithrootcauseanalysis

diagramsandrecommendationsthathasbeencompiledbythepeerassessmentteamstoreviewo Youshouldreviewthisreportforinaccuracies,itemsthatyoudonotagreewith,and

forhowrealistictherecommendationsareo ConsiderhowthisreportreconcileswiththeAARyourjurisdictionhasdrafted,

sharingthereportwiththosewhoattendedthemeeting,andifyoushouldreviseyourAAR

o Afterreviewingthereport,scheduleaconferencecallwiththeassessmentteamstodiscussanythingthatyoudonotagreewiththeassessmentteamson

Attemptsshouldthenbemadetoreconcilethisdisagreement Thereportshouldnotbefileduntiltherequestorsandassessmentteams

haveagreedonthecourseofthediscussionandappropriaterecommendations

Filingthereport.Afteryouandtheassessmentteamshaveagreedonthereport,root

causeanalysisdiagrams,andrecommendations,theassessmentteamswillsubmitthesedocumentstotheCIRsystem.

Followup.Additionalfollowupmayberequestedbythejurisdiction,CIRadministrators,oryourselfattheparties’discretion.


48

PreparingforapeerassessmentApproximately1Monthbeforethesitevisit: Schedulingsitevisits.IfCIRstaffavailabilityislimited,youandtherequestorswill

coordinateasitevisittime.Sitevisitswillideallybeatleastonefulldayandincludeathreepersonpeerassessmentteam. Tofindatimethatworksforallparties:

o Youwillbeprovidedthreepotentialdatesforsitevisitsandwillthenselectadatethatworksforyou.Youwillnotifytherequestorsthatthisdateworks

o Onceadateisselectedyouwillwaittobenotifiedofthemeetinglocationbytherequestors Shouldanovernightvisitbenecessary,youwillthenworkwith

requestorstofindahotelincloseproximitytothemeetinglocation Note:CIRstaffcanassistinhotelreservationsandtransportation

reservationswhenapplicable

Approximately1‐2weeksbeforethesitevisit: Documentreview.Priortothesitevisit,youwillreviewtheprefacetothepeer

assessmenttoolkit,thebackgrounddocumentonaCIRentry,anddocumentanyadditionalquestionsrelevanttothisevent. Additionaldocumentsthatmayberelevanttoreviewinclude:

o Relevantplans(tohelpassessmentteamsunderstandresponseactivitiesandidentifyopportunitiesforimprovement). Itemstoconsiderincludeincidentlogs,timelines,etc.

o Documentationofestablishedpartnershipsactivatedduringtheincident(toprovideadditionalcontext)

o Dataonrelevanthealthandsignificantnon‐healthoutcomes,responseactivities,etc.

o Mediareportso Timelineofmajoreventso Currentafteractionreports(shouldanyexist)o Otherrelevantdocumentsatthediscretionoftherequestingjurisdiction

One‐on‐OneDiscussionwithRequestors.Priortothesitevisit,youwilldiscussyourunderstandingoftheincidentwiththerequestorsofthereview.Thisdiscussionwillservetoestablish:

o Themosteffectiveon‐sitemeetingformat Areyoucomfortablewiththefacilitatedlookbackprocess(seetoolkit)? Doyouplantoconductoneononeinterviewaswell? Willtherebetimeforpre‐andpost‐debriefswiththerequestors?

o Keypointsfordiscussionduringyoursitevisit Doyouknowwhatcapabilitieswerestressedduringtheincident?

Appendix C Assessment Team Job Action Sheet


49

o Aroughattendancelistforthein‐personmeeting Therequestorswillhaveinvitedrelevantparties Youshouldconsider,areallkeypartiesrepresentedbasedonyour

documentreview?o Whatinformationthemeetingparticipantsareawareof

HasthedraftAARbeensharedwiththeseparties? Isthereanysensitiveinformationnottobediscussed?

PeerAssessmentactivitieson‐siteApproximately1hourbeforethemeetingbegins: Day‐ofmeetingpreparation.ThemeetingspaceshouldincludePowerPointdisplay,

boardsthatcanbewrittenonduringthediscussion,andaroundtablesetup.Youwillideallyassistrequestorsinthefollowingroomset‐up:o Chairsintheroomshouldbeorganizedsoparticipantscanfaceeachothero Thethreeassessmentteamsshoulddeterminetheirrolesforthemeeting

Oneofyouwillbethenotetaker Oneofyouwillbethediscussionfacilitator Oneofyouwillbefocusingontherootcauseanalysisprocessandprobing

thegrouponthisprocess Ifthreeassessmentteamsareon‐site,youmaydetermineamongyourselves

whowillfilleachroleo Tentcardsshouldbeon‐siteforparticipantswhoattendtowritetheirnameand

roleon.Tentcardsshouldthenbedisplayedateachindividuals’seat.Youshouldhaveatentcardaswell.

Meetingbegins: MeetingFacilitation.Thefollowingprocesshasbeenusedsuccessfullytofacilitatethe

in‐personlookback.AslidedeckhasbeenannotatedforyouandcanbefoundinAppendix3(linktotemplate).AsamplecompletedslidedeckcanbefoundthroughAppendix1,theWNVcasestudy.Itfollowsthebelowoutlinewithroughtimeestimates:o Agendao Objectivesformeetingo Corequestionstobediscussedtodayo Groundrulesfordiscussiono Introductionofparticipants

(Theaboveportionshouldtakeapproximately30‐45minutes)o Overviewofrootcauseanalysis,toexploreissuesfordiscussiono Rootcauseanalysisdiagramexplanation(seesectionbelowforfurtherdetails)o Rootcauseanalysisdiagramexamplefromyourexperienceo DOTMLPFinPHEPexplanation

o ExplainthisisanotherwaytoexamineRCAs(Theaboveportionshouldtakeapproximately30minutes)

o Reviewoftimelineofevents:o Probewhicheventsweretriggersforaction

o Figuresorvisualsrepresentingevent


50

(Theaboveportionshouldtakeapproximately15minutes)o Discussionoffirstissue(capabilityoridentifiedtopic):

Toinitiatetheconversation,youshouldconsiderprobingtheparticipantswithfollowing: Whowasinvolved Whywasthisanissue Whatfactorscontributedtothisbeinganissue Whatfactorswereinthedepartmentscontrol,whatfactorswere

unmodifiablebutplayedarole Astheassessmentteaminthefacilitatorroleyoushouldthenmoveonto

theRCAprocessthathasbeenintroduced: Definethestoryarc ConsidertheorganizationalgoalorobjectiveandrelevantPHEP

capability Identifytheresponders’responsechallengetofocuson Discussimmediatecausesofthatchallenge Discusscontributingfactorstothoseimmediatecauses

Duringthediscussion,ifyouaretaskedwithobservingrootcausesyoushouldattempttofillintheRCAdiagrambasedonthediscussion YoushouldthendisplaytheRCAdiagramyouhaveattemptedtoget

feedbackbeforemovingontothenextissue(Theaboveshouldtakeapproximately45minutesto1hour)

o Discussionofsecondissue(repeatforasmanyissuesastimefor)(Theaboveshouldtakeapproximately45minutesto1hour)(Consideralunchbreakhereforapproximately30minutes)

o LessonsLearnedfromdiscussionso Actionplanforthejurisdictiono Feedbackforyouo Yourcontactinformationtoprovideanyadditionalcomments

(Theaboveshouldtakeapproximately30‐45minutes)MeetingEndsafterapproximately5hours PostMeetingOn‐sitefollowup.Ifpossible,afterthemeetingyoushouldmeetwiththe

leaderswhorequestedtheassessmenttoaskforanyadditionalnotesandhowtheybelievethemeetingwent.Itisatyourdiscretionhowtohandlethisconversationandthenotetakershouldsharewiththerequestorsanythingthatwasunclearduringthediscussion

SiteVisitFollowUpWithinapproximately1monthaftersightvisit: AnalysisReport.Youwilldraftandshareananalysisreportwiththerequesting

jurisdiction.Thereportcontainsinformationgatheredthroughthefacilitatedlookbackand/orinterviews.Pleaseconsiderthefollowingwhendraftingyourreport:o Context:Basedonwhatyoureadpriortothesitevisit,andwhatyoulearnedduring

thesitevisit,thereportshouldstartwithadescriptionofthejurisdiction.


51

o Thiscontextshouldincludethejurisdictionalenvironmentthehealthdepartmentoperatesin(home‐rule,bystate,etc.),thenumberofjurisdictionsinvolved,informationaboutthelocationofthehealthdepartment,populationserved,andleadershipofotheragenciesthatwereinvolved(i.e.publicworks,emergencyresponders,agriculture,etc.).

o IncidentDescription:Thisshouldbeastraightfactpatternofwhathappenedduringtheincidentthatwasdiscussedduringthesitevisit.

o Re‐iteratingidentifiedrootcauses:Youshouldfollowtherootcauseanalysisdiagramandcounter‐factualapproachinidentifyingtherootcausesinthisincident.

ThisapproachisdiscussedinAppendix3(slides)andintheanalysissectionabove.

Formanyoftheissuesdiscussedduringthesitevisit,youwillnoticetherearesimilarrootcauses.

Includetheactualrootcauseanalysisdiagramsinyourreportasanappendix

o Makingrecommendations.Basedontheidentifiedrootcause,youshouldmakerecommendationsforchangingprotocolsandprocessesforthefutureattheendofyourreport.Theserecommendationsshouldbeinlinewithwhatwasdiscussedattheon‐sitemeeting,andcanalsodrawonyourown.Inmakingtheserecommendations,considertheseelements:

Whatlessonslearned/recommendationscanbegeneralizedusingthestoryarc?

Whatactionsmightpreventsimilarweaknessesorbuildonstrengthsinfutureresponses? Considertheactionsthedepartmentcouldhavetakentocreate

differentoutcomes Considerthealternativeactionsthedepartmentcouldhavetaken

Whatchangescouldbeimplementedpriortofutureeventstochangetheoutcomeslateron?

Shouldthegoalsbedifferent? Indifferentcircumstances,wouldthisresponsehavebeenappropriate? Whatdothelessonslearnedtellusabouttheunderlyingcapacityofthe

systemtorespondwhentheabovementionedcapabilitiesarestressed?Withinapproximately2monthsaftersightvisit: Discussingfindings.Youshouldsendyourfindings,whichincludeadescriptionofthe

incidentandtherootcausereporttotherequestingjurisdiction.Youshouldthenscheduleaconferencecallwiththerequestingjurisdictiontodiscussthereportandanythingthattherequestorsdonotagreewithyouon.Afterthiscall:o Attemptsshouldthenbemadetoreconcilethisdisagreement

Thereportshouldnotbefileduntiltherequestorsandassessmentteamshaveagreedonthecourseofthediscussionandappropriaterecommendations


52

Followup.Additionalfollowupmayberequestedbythejurisdiction,CIRadministrators,oryourselfattheparties’discretion.


53

EXAMPLE:TexasWestNileVirusoutbreakOnApril19thfrom9:30to2PMattheNorthCentralTexasTraumaRACoffices,600SixFlagsDrive,Arlington,Texas,wewillbeconductingareviewofthepublichealthresponsetoWestNileVirusin2012.ThegoalofthismeetingwillbetodiscussthefactorsthatcontributedtotheresponsechallengesindealingwithWestNileVirus,andwhatwerelearnedinthiscontext.Becauseyouwereanintegralpartoftheresponseandmitigationefforts,Ihopeyouwillconsiderattendingthismeetingtoshareyourperspective.Youmayalsowanttoinvitekeymembersofyourorganizationthatweredirectlyinvolvedintheresponseeffortsformosquitosurveillanceandcontrol.BecausetheresponsetoWestNileVirusinvolvedmanystateandlocalorganizations,thesuccessofthemeetinghingesonyourparticipation.Thisreviewisbeingorganizedinconjunctionwiththepeerassessmentlearningprogram.Thegoaloftheprogramandreviewaretoimprovetheprocessforafter‐actionlearningfromcriticalincidents.ThisprogramrecentlybeganwiththesupportofpublichealthresearchersandtheCDC.PleaseletXXX,knowifyouareabletoattendbycontactingherviaemail(XXX).Ifyouhaveanyadditionalquestionspleasefeelfreetocontact[ONEOFTHEUNDERSIGNED].Thanksverymuchandwehopetoseeyouthere,Director,ResponseandRecoveryUnitandDirector,CommunityPreparednessSectionDepartmentofStateHealthServices

Appendix D SAMPLE Meeting Invitation


54

Fortemplatespleasesee:http://www.hsph.harvard.edu/h‐perlc/preparedness‐toolkits/peer‐assessment LAMPS (Linking Assessment and Measurement to Performance in PHEP Systems) is the CDC-funded Preparedness and Emergency Response Research Center (PERRC) based at Harvard School of Public Health. This paper has benefited from the contributions of a practitioners advisory panel, practitioners at two field testing site, researchers associated with other PERRCs, and others. The authors would especially like to recognize the contributions of Jesse Bump and Elizabeth Lee of Georgetown University. This tool kit was developed with funding support awarded to the Harvard School of Public Health under cooperative agreements with the US Centers for Disease Control and Prevention (CDC) number 5P01TP000307-01.

Appendix E Additional Resources