Monitoring & Evaluation Using

Evidence Based Participatory Quality Improvement Techniques in

Public Health Programs

Presented by,

Mr. Rakesh. N. Pillai

Socio Research and Reform Foundation (NGO) - SRRF

1

STEPS TAKEN TO IMPROVE THE QUALITY OF A PUBLIC HEALTH PROGRAM

- AN EXAMPLE

Background

• India has 2nd leading global case load of Diabetes

• National prevalence is estimated at 8.3%*

• Diabetes Educators are not recognized as a health professional at the start of this program

India Diabetes Educator Program

2007-2011

*Source: International Diabetes Federation, Diabetes Atlas 5, 2011 2

Program Objective

• To develop a cadre of Diabetes Educators in India, creating a favorable environment to increase their acceptance in patient care

• To increase positive self efficacy of Diabetes Patients by promoting self care behaviour and achieve clinical metabolic control

Partnerships

- Society for Action in Community Health – SACH India

- Project HOPE, USA & Other implementing partners

- Eli-Lilly, BD, Bayers & Many Others

3

Program Overview

4

M&E Team - Objective

• To improve the Quality of the Program

• Set up an Evidence based Continuous Monitoring and Evaluation system

• Create a Health Management Information System (HMIS)

• Establish a mechanism for regular / quarterly evaluations and take corrective steps

• Overall to foster an environment for the success of the program

5

What is M&E ??? Note: M&E should be done at the planning phase

M&E

Document Program Activities

Summarize Figures

Establish Uniformity

Create Evidences

Disseminate Results

Generalize Results

Reduce Biases

6

QC – Seven Step Formula

Significance :

‘You can’t get good without knowing the basics’

QC- 7 step formula is the basic procedure for solving problems scientifically, rationally, efficiently and effectively.

This is a problem solving strategy which allows any individual to solve difficult problems rationally and scientifically

7

What is QC – Seven Step Formula Step Purpose Action Items

1 Understand Topic -Identify Problem -Decide topics to address

2 Understand Situation & Set targets

-Mechanisms for collecting information -Decide on targets (value and deadline)

3 Plan Activities -Decide ‘what to do’ -Decide schedule, Divide responsibilities

4 Analyse Causes -Check present values of characteristics -List possible causes, Items to tackle

5 Consider & implement counter measures

-Propose ideas for counter measures -Ways to put counter measures in place -Check details of counter measures

6 Check results -Compare results with targets -Identify tangible and intangible benefits

7 Standardize & Establish Control

Establish new standards & revise old, Decide on methods to control, Educate

8

Step 1 : Understanding the Topics – Diabetes Educator Program

» Check the roles of each departments & roles - Central Co-ordinating Office

- Finance & Admin - Program Implementing Team - Each Partner centres- MT’s & DE’s

» Check the policies & objectives of departments » List various problems by each departments » Select most appropriate problems to be tackled > Pareto Principle – 80% of problems could be

attributed to 20% of the causes Tools Used a. Cause & Effect Diagram b. Pareto Diagram 9

Components of the M&E System ?

M&E system should establish a system to;

1. map Performances of the Program

- through performance/outcome indicators

- during baseline, interim and final evaluation

2. map the Progress – through Process Indicators

- Quarterly reports

- Bi-annual reports etc

3. Detailed Implementation Plan (DIP)

4. Monitoring Cost & Time Schedules 10

» Decide on control characteristics

» Survey the past trends & existing levels

» Investigate the flow of work & Processes

» What sorts of changes occurred over time

» Stratify data as per problems

» Statistical dispersion / deviation is measured

» Prioritise

Tools Used:

Graphs, Checklists, Histograms, Scatter Diagram

Step 2 : Understand Situation & set targets – Diabetes Educator Program

11

We Strategized M & E as;

PDCA Wheel

Act Plan

Check Do

Source: TQM Approach / Kaizen Principle 12

We decided on;

a. Action Items

b. Schedule of activities

c. Activity Plan

Tools Used : Control Charts, Check lists etc.

Step 3 : Planning M&E – Diabetes Educator Program

13

Options considered for M&E

1. ‘Health Management Information Systems (HMIS)’

- Many software programs are available

2. Finally decided to make a simple, Excel sheet

- ‘Program Design Tool (PD Tool)’

14

Step 4: Analysis Questions to design ‘PD Tool’

According to

a. 5W1H [What, When, Where, Who, Why and How]

b. 4M Technique [Manpower, Machines, Materials and Methods]

• How many readings do we need?

• What period should we collect data?

• How to stratify data?

• What sampling or measurement methods?

15

Step 5: Design Counter Measures

16

Consider Problem from all sides

Collect ideas from experts

Collect ideas from various stakes in the processes

Using lateral thinking

Select countermeasure proposals

- Effect

- Feasibility

- Economy

‘PD Tool’ Definitions for Diabetes Educator Program

1. Performance Indicators

- High level indicators that lead directly to your Goal and Objectives

- Usually in Percentages

- Common for Baseline, Interim and Final

Objective Performance/Outcome Indicator

Foster an enabling environment for diabetes education and training in India

% improvement in wellbeing of Diabetes Patients

17

Definition’s & Examples ...Contd

2. Process Indicators

- Indicators that measure progress at the activity level.

- Usually simple counts

Program Specific Activities Process Indicators

Develop national curriculum for diabetes education

# of recognized curriculum developed

Conduct Master Trainer training

# of Master Trainers trained

18

Definition’s & Examples ...Contd

3. Detailed Implementation Plan (DIP)

- Based on Process Indicators / Activity based

- Map the progress every quarter through quarterly reports

Process Indicators

Target Year 1 Year 2 Year 3 Year 4

# Students certificate in Diabetes Education

3000 740 810 860 920

19

DIP System Hide blank rows

Target LOP

Year 2 Year 3 Year 4 Hide blank rows

Hide blank rows

Process Indicators

Q1 Q 2 Q 3 Q 4 Q1 Q 2 Q 3 Q 4 Q1 Q 2 Q 3 Q 4

Sep-08

Dec-08

Mar-09

Jun-09

Sep-09

Dec-09

Mar-10

Jun-10

Sep-10

Dec-10

Mar-11 Jun-11

Hide blank rows

1 # of Master Trainers trained

35 35 35

Hide blank rows Actual achieved 66 22 22 12

Hide blank rows

2

# of MT's actively engaged in conducting Diabetes Education and training 30

30

Hide blank rows Actual achieved 44

Hide blank rows

3

# Students enrolled in PG Certificate in Diabetes Education 4,325

4,325 325 500 500 500 500 500 500 500 500

Hide blank rows Actual achieved 4,542 304 464 482 480 470 500 411 448 450

Hide blank rows

4

# Students of PG Certificate in Diabetes Education presenting the Final Exam 3,000

3,000 197 80 90 100 100 200 200 547 581

Hide blank rows Actual achieved 2,855 197 80 90 100 100 200 200 547 581

20

Quarterly Report Reporting Period

Year 2011

Scoring Current Quarter green

Version 6.0, September 2010 Approvals

Expected to Date green Beneficiaries green

Project Name Diabetes Educator Project Country India 30-Dec-11 % Elapsed: 100% Project Code 05058

Goal Decrease morbidity and mortality related to diabetes in India Outcome Indicators

Indicator Baseline Year 4 Final Targe

t +/-

target +/- baseline

Data of data collection (planned or actual)

Date Result

30-Jun-11 1

average a1c for patients 30-Dec-

10 8

7.3

7.5 20% 80%

2

% improvement in well being index (WHO 5) 30-Dec-

10 61%

66%

71% -5% 5%

3

improvement in patients Self Efficacy score 30-Dec-

10 11

14

16 -200% 300%

Process Indicators Current Quarter

Expected to Date %

Description Target Actual

4 # of Master Trainers trained 0 0 189%

5 # of MT's actively engaged in training 0 0 147%

6 # Batches planned/initiated 0 0 99%

7 # Students enrolled i 0 0 105%

8 # Students presenting the Final Exam 0 0 103%

9 # Students graduating 0 0 103%

10 # Annual National Meeting 0 0 100%

11 # Partner Meetings 0 0 100%

12 # Abstracts accepted 0 0 100%

14

# Program evaluations conducted (mid/end term)

0 0 50%

15 % of students dropping out of the course 0% 0% 105%

Total Direct Beneficiaries 0 0 104% Expected to Date Direct + Indirect

213% Indirect Beneficiaries 0 0 214%

21

Timeline Set for the Program Activities Timeline

Project Name:

India Diabetes Educator Project

Version 3.0, September 2010 Country: India Start Date:

1-Jul-07 End Date:

Optional page

Activities Year 1 Year 2 Year 3

Program Specific Activities Q 1 Q 2 Q 3 Q 4 Q 1 Q 2 Q 3 Q 4 Q 1 Q 2 Q 3 Q 4

Sep-07

Dec-07

Mar-08

Jun-08

Sep-08

Dec-08

Mar-09

Jun-09

Sep-09

Dec-09

Mar-10

Jun-10

1 Develop national curriculum for diabetes education

2 Develop partnerships with leading centers for diabetes education, care and management

3 Develop regional network/TAG of Des

4 Conduct Master Trainer training

5 Master trainers to train Diabetes Educators

22

HMIS Developed

23

Hide blank rowsProgram Design Version 3.0, September 2010

Hide blank rows

Hide blank rows Location

Hide blank rows Start Date 1-Jul-07 $ award

Hide blank rows End Date 30-Dec-11

Hide blank rows

Hide blank rows

Hide blank rowsHide blank rows

Hide blank rowsFrequency

Hide blank rows1

Foster an enabling environment for diabetes education

and training in India1 average a1c for patients One time

Hide blank rows

2

Design and develop Diabetes Education distance learning

course for allied healthcare professionals benchmarked

on the best standards and incorporating Indian medical

practices

2 % improvement in well being index (WHO 5) One time

Hide blank rows

3

Increase knowledge, enhance education skills and

improve practice of allied healthcare professionals in

diabetes education, care and management

3 improvement in patients Self Efficacy score One time

Hide blank rows

4Improve training outcomes among people living with

diabetes managed by allied healthcare professionals

trained in diabetes education by Project HOPE

Hide blank rows

Hide blank rows

Hide blank rows

Hide blank rows1 Develop national curriculum for diabetes education 1 # of IDF recognized curriculum developed One time

Hide blank rows2

Develop partnerships with leading centers for diabetes

education, care and management2 # of centres adopting HOPE Diabetes Educator Program Annually

Hide blank rows3 Develop regional network/TAG of Des 3 # of regional network/TAG of Diabetes Educators formed Annually

Hide blank rows4 Conduct Master Trainer training 4 # of Master Trainers trained Baseline & final

Hide blank rows5

Master trainers to train Diabetes Educators5

# of MT's actively engaged in conducting Diabetes

Education and trainingSemi-annually

Hide blank rows6

Supervise/mentor Master Trainers6

# Batches planned/initiated for imparting HOPE PG

Certificate in Diabetes EducationSemi-annually

Hide blank rows7

Conduct annual MT meetings7

# Students enrolled in PG Certificate in Diabetes

EducationSemi-annually

Hide blank rows8

Conduct partner/donor meetings8

# Students of PG Certificate in Diabetes Education

presenting the Final ExamSemi-annually

Hide blank rows

9 Disseminate results 9

# Students graduating with HOPE's PG Certificate in

Diabetes Education with > = 70% grade in each milestone

and overall

Annually

IDEP Database

Guidelines and peers/

experts

IDEP Database

Data Source

IDEP Database

Frequency

Goal

Program Specific Activities Process Indicators

IDEP Database

IDEP Database

Donor Lilly, BD and Bayer

Objectives

IDEP Database

Problem Statement

Outcome Indicators

Networks/ MOU

Networks/ MOU

Decrease morbidity and mortality related to diabetes in India

Project Name

IndiaCountry

India Diabetes Educator Project

Diabetes is a global pandemic and India has the highest case load of any country. While patient education is key to promoting effective

self-management, no formal Diabetes Educator program exists at the national level. Poor self-management leads to increased morbidity

and mortality and diabetes is currently the second leading cause fo death nationwide.

Launch a sustainable model for forming allied health professionals as diabetes educators nationwide, in partnership with regional

centers of excellence.

Duration (# yrs)

Data Source

Program Snapshot

India

Evidences from

operations research

Evidences from

operations research

Evidences from

operations research

Step 6: Check Results

24

• Operational Research was Initiated

To evaluate the impact of HOPE-trained DEs on patient outcomes

a. Clinical parameters

b. Psychosocial parameters

c. Behavior change parameters

Results – Baseline to 6 Months Clinical Measures

Baseline

Mean (S.D)

6-Month

Mean (S.D) Significance

HbA1C 8.13 (1.54) 7.35 (1.15) < .0001

Fasting blood glucose (mg/dl) 148.7 (39.9) 130.4 (34) < .0001

Body mass index (BMI) 27.1 (4.5) 26.0 (4.1) < .0001

Total Cholesterol 174.7 (37.9) 163.4 (32.9) < .0001

Triglycerides 143.4 (74.3 ) 139 (55.8) .06

HDL Cholesterol 42.2 (11.5) 59.3 (28.2) < .0001

LDL Cholesterol 101.9 (31.2) 79.9 (36.8) < .0001

Blood

Pressure

Systolic BP

132.7 (13.8 )

129.1 (12.4) < .0001

Diastolic BP 80.9 (6.5) 79.7 (6.5) < .0001

* All analyses conducted using paired t-tests

Results – Baseline to 6 mos Psychosocial /Behavioral

Variables

Baseline

Mean (S.D)

6 mos

Mean (S.D) Significance

Well-being 61 (16.8) 65.7 (23.5) < .0001*

Self Efficacy 11.1 (3.5) 13.7 (3.6) < .0001*

% people eating 4+ fruits or

vegetable per day

14.8 % 43.9 % < .0001**

% people drinking 2+ litres of

water per day

58.1 % 70.2 % < .0001**

% people doing 30+ minutes of

exercise per day

22.8 % 71.9 % < .0001**

% people doing self monitoring 15.2 % 54.6 % < .0001**

* Analyses conducted using paired t-tests

** Analyses conducted using Chi Square

Conclusion

QC Seven Step Quality Improvement Technique is an useful tool to strategise Monitoring and

Evaluation Plans, especially in large, multicentre, Public Health Programs

Mastering this tools helps the Program Managers to Plan, Do, Check and Act upon the Objectives

finally achieving the Program Goals

27

Thank You

28