Critical appraisal Systematic R eview

Critical appraisal Systematic Review

กิ�ตต�พั�นธุ์� ฤกิษ์เกิษ์มภาควิ�ชาศั�ลยศัาสตร์

มหาวิ�ทยาล�ยเช�ยงใหม�

Systematic review vs Meta-analysis

• Systematic review: a systematic approach to minimising bias and error

• Meta-analysis: a statistical analysis, which aim to produce a single estimate of a treatment effect

• Systematic review may or may not include Meta-analysis

Why do we need systematic review?

• The early 1980s uncomplicated MI

“Should pt receive a beta-blocker for secondary prevention before discharge?”

Library: 4 randomised controlled trials (RCT)

Beta-blocker vs placebo

RCT 1.Mortality and hospital readmission is not different

RCT 2. Not conclusiveRCT 3. Beta-blocker not shown benefitRCT 4. Long term beta-blocker reduces the

mortality and rate of re-infarction

A review in BMJ 1981

• There is no clear evidence that beta-blocker improves long tem survival after MI despite almost 20 yrs of clinical trials

• Good enough!!

Another review in European Heart Journal 1981

“it seems perfectly reasonable to treat patients who have survived an infarction with beta-blocker”

Limitation of a single study

• Too small sample size false negative

Problem of Conventional review

• Prone to bias and error

– Select only evidence support the author’s view– Not specify methodological quality of studies– Finally choose most vote ignore sample size,

and design

Meta-analysis = combining all available data

• Attractive alternative to such large, expensive and problematic study

• Weight average of the result

large > small trial

Meta-analysis Beta-blocker trials - MI

Beta-blocker better Placebo better

Cumulative meta-analysis of beta-blocker trials

Cumulative meta-analysis

Significant effect from 1980 onwards (OR not across 1

Meta-analysis Beta-blocker trials - MI

Beta-blocker better Placebo better

Maybe Unnecessary trials

Benefit

• Estimate the overall effect

• Examine different result between studies (heterogeneity)

• Identified insufficient data

Cochrane collaboration• International organisation of

health care profession

• Promoting accessibility of systematic review

• Foster development of systematic review

• 50 collaborative review groups

www.cochrane.org/cochrane/ccweb.htm

Potentials of systematic review

• Good

• Bad

Systematic review Basic structure and types

กิ�ตต�พั�นธุ์� ฤกิษ์เกิษ์มภาควิ�ชาศั�ลยศัาสตร์

มหาวิ�ทยาล�ยเช�ยงใหม�

1. Basic structure

Like primary research

• Why- Introduction, background

• How-method

• What we found-result

• What it mean-discussion

Basic structure• Abstract • Introduction

– Background– Objectives

• Method “treat a paper like a patient in 1 reseach” – Type of studies– Inclusion criteria type of participants– Exclusion criteria– Type of intervention– Type of outcome measures– Search strategy for identification of study– Method of analysis

Basic structure

• Result (special diagram)• Conclusion• Reference

Odds ratio.05 .5 1 2

% Weigh t Odd s ra tio (95% CI)

0.32 (0.01,8.06) Foresell et al. 1989 23.2

0.13 (0.01,2.53) Sbarigia et al.1999 55.1

0.37 (0.01,9.17) Kasprzak et al. 1999 21.8

Pluskwa et al. 1989 0.0 Not estimable

Prough et al. 1989 0.0 Not estimable

Binder et al. 1999 0.0 Not estimable

0.22 (0.04,1.33) Overall (95% CI )

Black square = OR, horizontal line = 95% confidence interval

Area of black square = weight, diamond = combined OR with 95% CI

LA better GA better

Forest plot Stroke rate ‘LA vs GA in carotid sx trial’ 1966-2001

2. types

Types

• Systematic review of primary research– Observational studies– Diagnostic screening– RCT

3. Method “treat a paper like a patient in 1 reseach”

The process (1)

• Research question• Writing protocol• Searching• Article retrieval• Literature review

The process (2)

• Inclusion/ exclusion criteria• Validity and quality of articles• Data extraction/ synthesis• Interpretation

The question

• Is local anesthesia is better than general anesthesia during carotid endarterectomy?

Writing the protocol• Background• Objectives• Type of studies• Inclusion criteria• Type of participants• Exclusion criteria• Type of intervention• Type of outcome measures• Search strategy for identification of study• Method of analysis• Reference

Searching

• Medline• Other database• Hand searching the literature• Writing to people

Getting the article

• Which ones to get?• It takes time• Libraries• Inter-library Loans

Literature review

• You don’t have to read the whole paper yet!

• Translation

Validity and quality of articles

• Do read the paper and see what the author thought was wrong

• Unequal intervention/control size

• Hidden loss to follow up

Data extraction

• Read method carefully

• Design a form

Synthesis/ Interpretation

• Estimates and confidence intervals

pool effect make by statistic method e.g. Peto method (fix method) give more weight effect for large study than small study (P value)

• Difference between studies (Heterogeneity) Chi-squared test (P value)

Small RCTs show LA is marginal lower mortality than GA

Study or Subgroup

Binder 1999Forssell 1989Kasprzak 2006McCarthy 2001aPluskwa 1989Prough 1989Sbarigia 1999

Total (95% CI)

Total eventsHeterogeneity: Chi² = 1.44, df = 3 (P = 0.70); I² = 0%Test for overall effect: Z = 1.94 (P = 0.05)

Events

0001000

1

Total

27569134101355

286

Events

0111003

6

Total

19559533101052

274

Weight

14.5%14.5%28.5%

42.6%

100.0%

Peto, Fixed, 95% CI

Not estimable0.13 [0.00, 6.70]0.14 [0.00, 7.12]

0.97 [0.06, 15.85]Not estimableNot estimable

0.12 [0.01, 1.21]

0.23 [0.05, 1.01]

Local General Peto Odds Ratio Peto Odds RatioPeto, Fixed, 95% CI

0.10.2 0.5 1 2 5 10Local better General better

Critical Appraisal

1. Are the result valid?

2. What are the results?

1. Are the result valid?

• Did this review address a sensible clinical question

• Was the search for relavant studies detailed and exhaustive?

• Were selection and assessment of studies reproducible?

• Were the primary studies of high methodological quality?

Publication bias

“A (significant) beneficial treatment effect are published, but an equal result remain unpublished”

• In general medical journal and public heath journal reported statistically significant 85.4%

• In psychological journal 95.6%

Time lag bias

• “Positive result will dominate the literature for several year until the negative will report later”

• HIV trial in USA, median time to publish of positive result 4.2 years, but negative result 6.4 years

Duplicate publication bias• “ one study presents and reports

several times” “ include this lead to overestimation of treatment effect”

• Ondersetron to prevent postoperative nausea vomitting 16 studies

3 duplicated papers

• Sometimes difficult to say, since not share single common authors !!!!!!

Language bias

• “Authors tend to report positive result in international papers, English language journal, but if negative result are published in local journal”

Outcome reporting bias

• In trials many outcome is recorded but only favorable finding will be reported

• Clinical trials by drug companies, unpublished trials gave information on adverse effect > published trials

Selection Bias

• Tend to happen in non Randomised controlled trials (non RCT)

• For example select low risk group to new treatment group

2. What are the result?

• Were the results similar from study to study? If yes, the credit of single estimates is OK.– Point estimates similar?– Overlapping confidence interval– Test for heterogeneity? (Chi square test)– Percentage of variability (I2 ) good < 20%,

concern 20-50%,serious concern > 50%

Look overlapping confidence

interval

Study or Subgroup

Agrifoglio 1987Allen 1994Andersen 1980Bartoloni 1991Becquemin 1991Bowyer 2000Brown 1999Calligaro 2001Corson 1987ECST 1998Gabelman 1983Ghali 1997Godin 1989Gurer 2003Harbaugh 2000Hartsell 1999Imparato 1998Jopling 1983Kalko 2007Kraiss 1995Kucey 1998Love 2000Marrocco 2004McCarthy 2001bMcCleary 1996Mertens 2000Mofidi 2006Muskett 1986NASCET 1998Ombrellaro 1996Palmer 1989Quigley 2000Reina 1998Rignano 1999Santamaria 2004Schwartz 1988Shah 1994Sternbach 2002Stone 2000Stoughton 1998Syrek 1999Taylor 1999

Total (95% CI)

Total eventsHeterogeneity: Chi² = 33.76, df = 30 (P = 0.29); I² = 11%Test for overall effect: Z = 2.65 (P = 0.008)

Events

0260120310000122

470200002101000203000520012

88

Total

2927417969

13227215718514558483350

200632116

338250

30618

18520050

1003222

19230

10414018412611676

20314965422667

15083

140

9564

Events

241062043

17100201

12031

233000030

14001312231000

16

131

Total

2831015275

220228132216223

1681405750

16517111859330

127178

109624363

1403372

17945

13101263778

18816956

162219324615826

1152

10631

Weight

1.5%4.4%5.1%

4.8%3.0%

5.1%2.8%1.6%0.7%

2.2%1.0%2.2%

22.2%

3.1%0.3%8.4%2.2%

1.5%0.8%

3.0%

2.8%

0.8%0.7%4.2%0.6%1.0%1.5%4.5%2.2%

0.5%5.2%

100.0%

Peto, Fixed, 95% CI

0.13 [0.01, 2.06]0.58 [0.12, 2.89]

3.66 [0.82, 16.41]Not estimable

0.37 [0.08, 1.71]0.84 [0.12, 6.02]

Not estimable0.87 [0.20, 3.91]0.54 [0.07, 4.08]0.35 [0.02, 5.03]0.11 [0.00, 5.68]

Not estimableNot estimable

0.42 [0.04, 4.09]3.57 [0.12, 105.59]1.99 [0.21, 19.35]0.65 [0.32, 1.33]

Not estimable0.22 [0.03, 1.55]

0.33 [0.00, 294.58]0.30 [0.09, 0.98]0.16 [0.02, 1.56]

Not estimable11.13 [0.67, 186.25]7.62 [0.15, 384.38]

Not estimable0.34 [0.05, 2.43]

Not estimable0.34 [0.04, 2.53]

Not estimable3.34 [0.08, 137.98]

0.07 [0.00, 4.13]1.67 [0.32, 8.77]

0.23 [0.00, 16.24]0.01 [0.00, 0.28]0.15 [0.01, 2.35]0.51 [0.10, 2.56]

2.89 [0.29, 28.92]Not estimableNot estimable

3.72 [0.04, 369.48]1.03 [0.23, 4.60]

0.63 [0.45, 0.89]

Local General Peto Odds Ratio Peto Odds RatioPeto, Fixed, 95% CI

0.10.2 0.5 1 2 5 10Local better General better

Rerkasem Cochrane Database Syst Rev 2008; (4):CD000126.

Small RCTs show LA is marginal lower mortality than GA

Study or Subgroup

Binder 1999Forssell 1989Kasprzak 2006McCarthy 2001aPluskwa 1989Prough 1989Sbarigia 1999

Total (95% CI)

Total eventsHeterogeneity: Chi² = 1.44, df = 3 (P = 0.70); I² = 0%Test for overall effect: Z = 1.94 (P = 0.05)

Events

0001000

1

Total

27569134101355

286

Events

0111003

6

Total

19559533101052

274

Weight

14.5%14.5%28.5%

42.6%

100.0%

Peto, Fixed, 95% CI

Not estimable0.13 [0.00, 6.70]0.14 [0.00, 7.12]

0.97 [0.06, 15.85]Not estimableNot estimable

0.12 [0.01, 1.21]

0.23 [0.05, 1.01]

Local General Peto Odds Ratio Peto Odds RatioPeto, Fixed, 95% CI

0.10.2 0.5 1 2 5 10Local better General better

Rerkasem Cochrane Database Syst Rev 2008 (4):CD000126.

2. What are the result?

• Were the results similar from study to study?

• What are the overall results of the reviews?

• How precise were the results?