Upload
kathrin-dentler
View
390
Download
1
Embed Size (px)
DESCRIPTION
In order to be able to automatically calculate clinical quality indicators, we have proposed CLIF, a stepwise method for clinical quality indicator formalisation. Quality indicators are used for external accountability and hospital comparison. As clinical quality indicators are computed in a decentralised manner by the hospitals themselves, reproducibility of the formalisation method is essential to ensure the comparability of calculated values. Thus, we performed a case study to investigate the reproducibility of CLIF. Eight participants formalised the same sample quality indicator with the help of a web-based indicator-authoring tool that facilitates the application of CLIF. We analysed the results per step and concluded that the method itself leads to reproducible results. To further improve reproducibility, ambiguities in the indicator text must be clarified and trained experts are needed to encode clinical concepts and to specify the relations between concepts.
Citation preview
1 / 15
The Reproducibility of CLIF, a Method for Clinical Quality
Indicator Formalisation Kathrin Dentler, Ronald Cornet, Annette ten Teije, Kristien Tytgat, Jean Klinkenbijl
and Nicolette de Keizer
2 / 15
Quality Indicators Used • internally and • externally Ø need to be
well-formalised to lead to comparable results
Ø CLIF Ø needs to be
reproducible
3 / 15
Case Study
Ø We performed a case study to investigate CLIF’s reproducibility Ø Developed reference standard together with experts for sample indicator
Ø 8 participants (Medical Informatics Master Students) formalized sample indicator with the help of a web-based tool
4 / 15
Employed Indicator
Numerator: Number of patients who had 10 or more lymph nodes examined after resection of a primary colon carcinoma. Denominator: Number of patients who had lymph nodes examined after resection of a primary colon carcinoma. - Exclusion criteria: Previous radiotherapy and recurrent colon carcinomas
Evidence-‐‑based (correct staging leads to beHer outcome), requires data from several sources
5 / 15
CLIF’s 8 Steps (construct two queries)
1) Encode relevant concepts in terms of a terminology
2) Define the information model 3) Formalise temporal constraints 4) Formalise numeric constraints 5) Formalise Boolean constraints 6) Group constraints by Boolean connectors 7) Formalise in- and exclusion criteria 8) Construct the denominator by removing
constraints that only aim at the numerator
6 / 15
Step 1: Encode relevant concepts in terms of a (standard) terminology
Numerator: Number of patients who had 10 or more lymph nodes examined after resection of a primary colon carcinoma. Exclusion criteria: Previous radiotherapy and recurrent colon carcinomas
7 / 15
0 1 2 3 4 5 6 7 8 9
lymph nodes examined
resection of colon carcinoma
radiotherapy
primary colon carcinoma
recurrent colon carcinoma
Users’ Results Step 1
Examination of lymph nodes
Colectomy
Primary malignant neoplasm of colon
Carcinoma of colon
Radiation oncology AND/OR radiotherapy
Fleiss'ʹ kappa: 0.754 (substantial agreement)
8 / 15
Step 2: Define the Information Model
Numerator: Number of patients who had 10 or more lymph nodes examined after resection of a primary colon carcinoma. Problem-oriented information model: relate all procedures to diagnoses. Ø Users had major difficulties with relating concepts in
the information model.
9 / 15
Step 3: Formalise Temporal Constraints
Numerator: Number of patients who had 10 or more lymph nodes examined after resection of a primary colon carcinoma. Exclusion criteria: Previous radiotherapy and recurrent colon carcinomas Reporting year: 2010 Ø Users had difficulties due to ambiguities: o Which procedure during the reporting year? Lymph
node examination or resection? Or both? o Before which event should the radiotherapy have taken
place?
10 / 15
Step 4: Formalise Numeric Constraints
Numerator: Number of patients who had 10 or more lymph nodes examined after resection of a primary colon carcinoma. Ø All users met the reference standard
11 / 15
Step 5 & 6: Boolean Constraints & Connectors (not applicable)
12 / 15
Step 7: Exclusion Criteria
Exclusion criteria: Previous radiotherapy and recurrent colon carcinomas
Ø Only consecutive errors: constraints / concepts have not been defined previously (step 1 and 3) and thus could not be excluded.
13 / 15
Step 8: Difference between Numerator and Denominator
Numerator: Number of patients who had 10 or more lymph nodes examined after resection of a primary colon carcinoma. Denominator: Number of patients who had lymph nodes examined after resection of a primary colon carcinoma. Ø All users met the reference standard
14 / 15
Overview of Users’ Results Ad
here
nce
to R
efer
ence
Sta
ndar
d
conc
epts
inf. m
odel
tempo
ral
numeri
c
exclu
sion
num/de
nom
0
20
40
60
80
100
●
participant 1participant 2participant 3participant 4participant 5participant 6participant 7participant 8
●
●
●
● ● ●
15 / 15
Recommendations, Conclusions and Future Work
Ø We recommend indicator-releasing organisations to publish indicators together with sets of concepts and to formulate indicators as precisely as possible - especially temporal relations. Otherwise: compare with caution!
Ø Those responsible to calculate indicators must be trained in the employed information model.
Ø CLIF helps to make ambiguities in indicators explicit and
can support reproducible results. Ø Future work: Investigate the generalizability of CLIF and
the use of standard information models.