An Introduction to Critical Appraisal

Laura WilkesTrust Librarian

West Suffolk Foundation Trust

Laura.Wilkes@wsh.nhs.uk

Last updated May 2012

Learning OutcomesBy the end of this session you will:

o Understand what Critical Appraisal iso Be aware of some of the different types of researcho Be able to interpret basic statistics within a research

papero Gain experience in critically appraising a research

paper

Evidence Based Practice“"Evidence-based

medicine is the conscientious, explicit and judicious use of current best evidence in making decisions about the care of individual patients.”

Sackett, 1996

Hierarchy of Evidence

Types of ResearchSystematic Reviews

A literature review focused on a single question which tries to identify, appraise, select and synthesize all high quality research relevant to that question.

Combines results of several RCTs or other types of evidence Systematic reviews follow a clear sequence of steps:

Defining an appropriate question Searching the literature – published and unpublished, English &

non-English Assessing the studies – involves 2 independent reviewers Combining the results & producing a “bottom-line” Placing the findings into context

Advantages Limits bias Good quality evidence Added power by synthesising individual study results

Types of ResearchExperimental:Randomised Controlled Trials (RCTs)

Randomly assign individuals to an intervention group or a control group in order to measure the effectiveness of an intervention.

Gold standard for treatment evaluationsHowever, some studies are not suitable for RCTs

Types of ResearchObservational StudiesCohort Studies

A non-experimental study design Follows a group of people (a cohort), and then looks

at how events differ among people within the group.Follow up period can be years or decadesProspective cohort studies (which track participants

forward in time) are more reliable than retrospective cohort studies

Can be expensive to conduct

Types of ResearchObservationalCase-Control Study

Examines a group of people who have experienced an event (usually an adverse event) and a group of people who have not experienced the same event, and looks at what risk factors both groups have been exposed to

Retrospective, therefore prone to recall bias, but quick & involve small numbers

Primary method of studying new or unusual outcomes

Case-SeriesAnalysis of series of people with the disease there is no comparison group

Critical AppraisalThe process of systematically weighing up the

quality and relevance of research to see how useful it is in decision making

It is the balanced assessment of benefits and strengths of research against its flaws and weaknesses

Increases the effectiveness of your readingIt can help you make informed decisionsIs a skill that needs to be practised

How do I Appraise?• You don’t need to be a statistics expert• Ready-made checklists help you focus on the

most important aspects of the articleDifferent checklists available for different types of

research (RCTs, systematic reviews, case-control studies, etc).

Checklist for Qualitative researchAvailable free from CASP

http://www.casp-uk.net

Critical Appraisal Critical appraisal of any study design must assess:Validity

Were sound scientific methods used? Chance / Bias / Confounding Factors

ResultsWhat are the results and how are they expressed?

RelevanceAre the findings generalisable – can they be applied to

settings / situations outside the research study? Do these results apply to my local context?

Potential Errors of ResearchChance

A random error appearing to cause an association between an intervention and an outcome. Probability of a random error is estimated using statistics (p values & confidence intervals)

BiasThe deviation of results from the truth due to systematic error in the

research methodology Selection bias – when two groups being studied differ systematically in

some way Observer bias – where there are systematic differences in the way groups

are treated or in how information is collected

Confounding Factors An error of interpretation Where part of the observed relationship between two variables is due to the

action of a third variable Known confounders: e.g. age, gender, smoking, etc.

Critical Appraisal of an RCTScreening questions:

1. Did the study ask a clearly focused research question? Patient / Intervention / Outcome Is it relevant to you?

2. Did the authors use an appropriate research method? Is an RCT the most appropriate?

If the answer to these questions is “YES” you can carry on with the rest of the checklist!

RCT Appraisal Checklist continuedDetailed Questions:3. Were participants appropriately allocated to groups?

Sample: Is it representative of the target population (the population to which the results

will be applied) Convenience sampling? Exclusion / Inclusion criteria – do you agree?

If too selective results may not be generalisable Bias?

Are the groups balanced? Important that groups are similar at the beginning of a trial so there is more

chance of differences at the end being due to the intervention Look at the Baseline Characteristics Table

Was group allocation truly random? Bias can occur if patients, carers, or researchers can influence allocation

Randomisation Randomisation ensures individuals have an equal chance of being

allocated to any Group

Potential confounding factors will be equally distributed between groups

Successful randomisation requires that group allocation cannot be predicted in advance – allocation concealment avoids bias

Allocation concealment ensures all those involved in the trial are unable to predict the allocation of the next participant until that participant is enrolled. Methods include telephone randomisation, or using consecutive sealed opaque envelopes.

A good study should indicate who generated the randomisation sequence, the method used, and how concealment was achieved & monitored

Randomisation Methods: Computer Generated Sequence

E.g.:4,8,3,2,7,2,6,6,3,4,2,1,6,2,0,…….

Two Groups (criterion: even-odd): AABABAAABAABAAAThree Groups: (criterion:{1,2,3}~A, {4,5,6}~B, {7,8,9}~C; ignore

0’s) BCAACABBABAABA……

Permuted Block RandomisationUsed for small studies to maintain reasonably good

balance among groupsWith a block size of 4 for two groups (A,B), there are 6

possible permutations and they can be coded as:1=AABB, 2=ABAB, 3=ABBA, 4=BAAB, 5=BABA, 6=BBAA Each number in the random number sequence in turn

selects the next block, determining the next four participant allocations (ignoring numbers 0,7,8 and 9).

e.g., The sequence 67126814…. will produce BBAA AABB ABAB BBAA AABB BAAB.

Stratified Block RandomisationA set of permuted blocks is generated for each

combination of prognostic factorsE.g. age group, severity of condition, and

treatment centre. Stratification means having separate block

randomisation schemes for each combination of characteristics (‘stratum’)

For example, in a study where you expect treatment effect to differ with age and sex you may have four strata: male over 65, male under 65, female over 65 and female under 65

Questions continued4. Were participants, staff and study personnel “blind” to

participants study group? Withholding information about the assigned interventions from

people involved in the trial who may be influenced by this knowledge

Eliminates error through bias Is blinding always possible? Different levels of Blinding

Single- blind Trial Either the researcher or the subject is blind to the allocation

Double-blind Trial Both researcher and subject are blinded

Triple-blind Trial Subject, researcher and analyst

5. Were all the participants who entered the trial accounted for at its conclusion? Loss to follow up?

A common cause of missing data, especially in long-term studies How important are the losses?

5% probably OK but >20% poses threat to validity Losses equally distributed?

Intention to treat analysis? Analysing people at the end of the trial, in the groups to which they were randomised, even if

they did not receive the intended intervention Maintains randomisation Prevents bias caused by loss of participants

6. Were all the participants in all groups followed up and data collected the same way? Were groups treated equally? Followed over the same time period / received same attention Bias?

Differences in the way data collected, measurements taken etc. if individuals know which group they have been allocated to

7. Did the study have enough participants to minimise the play of chance?

Before calculating the sample size, a clinically significant treatment effect is estimated

POWER Calculation: calculates how large the sample should be in order to have a high chance of detecting a true difference between the groups.

Avoids a type I (false positive) or type II (false negative) errorCalculated before the study beginsLook for 80% - 90% power –probability of finding a

significant difference with a given sample sizeSample size increases when a small treatment effect is

expected & with higher power.

Are the Results Significant?8. How are the results presented and what is

the main result?

What sort of data have they got & have they used appropriate statistical tests?

Are the results expressed in terms of likely harm or benefit? Relative Risk Numbers Needed to Treat

How meaningful is the result?

Event Rates

Number of people experiencing an event as a proportion of the number of people in the population

• Form the basis of other calculations Control Event Rate (CER) Experimental Event Rate (EER)

Emerg Med J 2008 25: 26-29:

Proportion with recurrent headache (whole sample) CER = 12/31 = 39% EER = 8/30 = 27%

Risk of benefit and harmRelative Risk (RR) = compares the risk in 2 different groups

of peopletells us how many times more likely it is that an event will

occur in the treatment group relative to the control group EER / CER Relative Risk of 1 means the risk is the same in each group <1 = treatment reduces risk of event >1 = treatment increases risk of event

27/39 = 0.69 = treatment reduces risk of event

Risk of headache is 0.69 times lower in the treatment group than in the control group.

Risk continuedAbsolute risk reduction (ARR)

Difference in risk between experimental and control groups

Risk of Event in Control Group – Risk of Event in intervention groupARR=0 Treatment has no effectARR positive – Treatment is beneficialARR negative – Treatment is harmful

39% - 27% = 12%Dexamethasone reduces the absolute risk

of recurrent headache by 12%

Relative Risk Reduction (RRR)tells us the reduction in the rate of the outcome in the

treatment group relative to that in the control group

ARR / CER Or 1 – RR0.12 / 0.39 = 0.31 = 31%1-0.69 = 0.31 = 31%Dexamethasone reduces the risk of recurrent

headache by 31% relative to that occurring in the control group.

Absolute Risk Reduction & Relative Risk Reduction

Results of hypothetical trial of a new drug for myocardial infarction

Compare with Low Risk Patients

Later studied in a lower risk population: 10% mortality rate at 30 days

among untreated 7.5% mortality among treated

Absolute Risk Reduction is therefore 2.5% 10% - 7.5% = 2.5%

Relative Risk Reduction is 25% 2.5 / 10 = 0.25 = 25%

Relative Risk Reduction is often more impressive than Absolute Risk Reduction

The lower the event rate the smaller the absolute risk reduction

Odds Ratio Odds = number of

events / number of non events

Odds Ratio = odds in treatment group / odds in control group

If odds is greater than 1 then the event (outcome) is more likely to happen than not.

Headache

No headache

Odds

Treatment

8/31 8 23 0.35

Placebo 12/32

12 20 0.6

Odds Ratio = 35/60 = 0.59 Odds of recurrent headache almost 50% less in the treatment group

Numbers Needed to TreatMeasures the impact of a treatment or interventionStates how many patients need to be treated in

order to prevent an event which would otherwise occur.

NNT = 10 means that 10 patients need to be treated to prevent one adverse outcome

The closer to 1 the betterCalculation:

1 / ARR (if ARR expressed as a proportion)100/ARR (if ARR expressed as a %)

100/12 = 8

Results9. How precise are the results?P Values

P=ProbabilityA p-value is a measure of statistical significance

which tells us the probability of an event occurring due to chance alone

P values only from 0 to 1

If P Value is very small (e.g. P<0.001) the result is unlikely to be due to chance (1 in 1000)

Generally, look for P<0.05 (1 in 20)

In simple terms, probability (p-value) can only take values between 0 and 1:

0|-----------------------|--------------------|1

Impossible…....... Absolutely certain…

If p=0.001 the likelihood of a result happening by chance is extremely low: 1 in 1000

If p=0.05 it is fairly unlikely that the result happened by chance 1 in 20

If p=0.5 it is fairly likely that the result happened by chance 1 in 2

If p=0.75 it is very likely that the result happened by chance 3 in 4

P - values

Confidence IntervalsAn alternative way of assessing the effects

of chanceThe result of the trial is a “point estimate” –

if you ran the trial again you will get a different result

The Confidence Interval gives the range in which you think the real answer

The 95% CI is the range in which we are 95% certain that the true population value lies

Look at how wide the interval is, and the values at each end

E.g. RR = 0.69 95% CI 0.33 to 1.45

Forest Plot – Simple Example

Individual sample size

Combined Results

Confidence Interval

Line of No Effect

Best Estimate

The shorter the Confidence Interval (CI) the more confident we can be that the results are true

If the CI crosses the line of no effect, then the results of that study are not statistically significant

Line of No Effect = 1 for Relative Risk or Odds Ratio0 for Mean resultsFavours Treatment Favours Control

Relevance10. Were all important outcomes considered so

the results can be applied?Does your local setting / population differ from that

in the research in ways that would produce different results?

Can you provide the same treatment in your local setting?

Does any benefit reported outweigh any harm?Should policy or practice change as a result of this

research?

Is there a Systematic Review on the subject? …… Check the Cochrane LibraryNo Cochrane Reviews, but 2 “Other Reviews” – Systematic

Reviews but not produced by the Cochrane Collaboration.

ConclusionCritical Appraisal is part of Evidence Based

HealthcareIt takes practiceUse CASP checklistsDepth of Appraisal is your choiceOnly you can assess usefulness

Useful websiteswww.healthknowledge.org.uk/interactive-learning/

finding-and-appraising-the-evidencewww.thennt.com/www.casp-uk.net/www.wikipedia.org very good for a basic guide to

statistics in critical appraisal. Search for critical appraisal.