Basic Concepts of Research

Basic Components

Objectives Search process QUORUM diagram Test of heterogeneity and model used for

meta-analysis Types of data and effect size Risk of biases assessment Forest Plot

Objectives

A statement of questions being addressed with reference to

[P = participants/populations] [I=interventions] [C=comparisons] [O=outcomes], and [S = study design]

PICO Questions

P = Population/Problem I = Intervention/indicator/

index test C = Comparator O = Outcome

P = Population

Population What are the important characteristics

that define the participants? What is the setting? (hospital,

community, etc) How is this disease/condition defined? Who make the diagnosis?

P = Population

Population Are there any co-morbidities to be excluded? Are there any other types of people that

should be excluded or considered in the review (because of the intervention in a different way)?

How will studies involving only a subset of relevant participants be handled?

I = Intervention

Intervention Does the intervention have variations (e.g.

dosage, components, mode of delivery, personnel, frequency, duration or timing)?

Are all variations to be included (e.g. is there a minimum dose or components without which intervention may not be expected to work in the same way)?

C = Comparison

Comparison What are you interested in comparing the

intervention to (e.g. an active intervention, no intervention or placebo, any available comparison)?

Refer back to primary question of review

O = Outcomes

Outcomes What are the important outcomes that you

plan to measure in your review? What are your primary outcomes (max 3)? What are your secondary outcomes? Have you included possible adverse effects?

O = Outcomes

Outcomes How should the outcomes be measured (e.g.

validated tools)? Are there important time points at which

outcomes should be measured (e.g. long enough to expect an observable effect)?

Have you included outcomes relevant to all potential decision-makers?

CRITICAL APPRAISALS

STUDY QUESTION

CRITICAL APPRAISAL

P= Population R = Recruitment I= Intervention A = Allocation

C = Comparison M = Maintenance O = Outcome M = Measurement

B = Blinding O = Objective outcomes

CRITICAL APPRAISAL

Study question: Population Critical appraisal: recruitment Were the subjects representative of the target

population? Larger studies are preferred.

For continuous outcomes (e.g. BP, weight, etc) ! 50 100 might be sufficient

For binary outcomes (e.g. AMI) ! numbers depend on how common it is.

Common events ! hundreds would be enough. Rare events ! may require thousands to capture the events

CRITICAL APPRAISAL

Study question: Intervention Critical appraisal: Allocation

It is vital that the groups are matched as closely as possible in every way except for the intervention (or exposure or other indicator).

CRITICAL APPRAISAL

Study question: Intervention Critical appraisal: Allocation

If the groups are not comparable to begin with, then any difference in outcomes may be due to one of the non-matched characteristics (or confounding factors), rather than due to the intervention (or exposure or other indicator) under consideration.

CRITICAL APPRAISAL

Study question: Intervention Critical appraisal: Allocation For example, ways in which groups could differ:

Age Gender Smoker/non-smoker Disease severity Other risk factors, etc The most important matching factors are those that could

predict the outcomes of the condition, which are often most related to the severity of the illness

CRITICAL APPRAISAL

Study question: Intervention Critical appraisal: Allocation Blinding process: must ensure that neither

the trial subjects nor the investigators can influence the group each person ends up in (allocation concealment)

Otherwise, this could introduce selective allocations, skewing the groups, or bias in reporting of outcomes

CRITICAL APPRAISAL

Study question: Comparison Critical appraisal: Maintenance Once comparable groups have been set up,

it is important that they stay that way. The study groups should be managed so

that the only difference between the groups is the factor being tested (e.g. Drug A vs no drug A, smoking vs no smoking).

CRITICAL APPRAISAL

Study question: Comparison Critical appraisal: Maintenance Groups should be treated exactly the same

as the experimental group in every respect except for the factor being tested.

Also, ensure identical measurement strategy equipment, methods, assessors

CRITICAL APPRAISAL

Study question: Comparison Inevitably, some subjects do drop out or lost

during follow-up. This can be serious when the remaining groups

may no longer be comparable. Therefore, check that:

Subjects at the start = subjects at the end (majority has been accounted for)

Subjects are analyzed in the groups that they started out in (the intention-to-treat principle)

Intention-To-Treat

The Intention-to-Treat principle Once a subject is randomized, he or she

should be analyzed in the group they are randomized to, even if they never receive the treatment, discontinue the trial or cross over to the other groups.

CRITICAL APPRAISAL

Study question: Comparison Critical appraisal: Maintenance The results can also be biased if the subjects

are not followed up for long enough for relevant outcomes to be revealed in both groups.

It is therefore important that the subjects are followed up until the relevant outcome occurs or until death (cohort studies).

CRITICAL APPRAISAL

Study question: Outcome Critical appraisal: Measurement Were the outcomes measured with: Blinded subjects and assessors and/or Objective outcomes? Measurement bias can be reduced if the

outcomes measured are objective (e.g. BP) rather than subjective (e.g. feeling better)

The EQUATOR Network

http://www.equator-network.org/

The EQUATOR Network

PRISMA Statement

Search process

Where do the information sources come from?

For each database searched, authors should report the database, platform, or provider (such as Ovid, Dialog, PubMed) and the start and end dates for the search of each database

What are the search terms?

Example

Studies were identified by searching electronic databases, scanning reference lists of articles and consultation with experts in the field..No limits were applied for language and foreign papers were translated. This search was applied to Medline (1966 Present), CancerLit (1975 Present), and adapted for Embase (1980 Present)..and Pre-Medline electronic databases. Cochrane and DARE (Database of Abstracts of Reviews of Effectiveness) databases were reviewedThe last search was run on 19 June 2001. In addition, we handsearched contents pages of Journal of Clinical Oncology 2001, together with abstracts printed in these journals 1999 - 2001.

QUORUM diagram

The Quality of Reporting of Meta-Analyses (QUORUM) guidelines contain a listing of essential steps and items that must be included in a well-conducted meta-analysis

QUORUM diagram

Test of Heterogeneity

Heterogeneity is a term used to describe variability among studies

Two sources of heterogeneity in a meta-analysis A. Within-study variability: Due to sampling variability. Always present because every study uses different samples


B. Between-studies variability Is the true heterogeneity Influenced by numerous characteristics that

vary among the studies (e.g., variations in study design, variations in treatment, etc)


When the studies results only differ by the sampling error (homogeneous case) ! a fixed-effects model is used

When the study results differ by more than the sampling error, which means, both within- and between-studies variability (heterogeneous case), ! random-effects model is used

Types of Data and Effect Size

Type of data Type of effect size Continuous data Mean difference

Standardized mean difference

Dichotomous data Odds ratio Risk ratio/relative risk

Classifications of biases

Type of bias Description

Selection bias Systematic differences between baseline characteristics of the groups that are compared

Performance bias Systematic differences between groups in the care that is provided, or in exposure to factors other than the interventions of interest.

Detection bias Systematic differences between groups in how outcomes are determined.

Attrition bias Systematic differences between groups in withdrawals from a study.

Reporting bias Systematic differences between reported and unreported findings.

http://www.cochrane.org/handbook

No matter how carefully a sample is selected to be a fair and unbiased representation of the population,

relying on information from a sample will always lead to some level of uncertainty. CI expresses this

uncertainty.

Why Confidence Interval?

Narrow vs Wide CI

Generally when a confidence interval is very wide, it is an indication of an inadequate sample size

A narrow confidence interval (CI) implies high precision; we can specify plausible values to within a tiny range.

A wide confidence interval implies poor precision; we can only specify plausible values to a broad and uninformative range.

Why Confidence Interval?

Important Question:

Does the interval contain a value that implies no change

or no effect?

When CI contains Null Value

With a confidence interval for a difference look to see whether that interval includes zero.

With a confidence interval for a ratio, look to see whether that interval contains one. Null Value

Red line : CI that contains the null value implying no statistically significant change. Blue and Black lines: statistical significant effects

Forest Plot

A forest plot displays effect estimates and confidence intervals for both individual studies and meta-analyses. Each individual study is represented by a square at the point estimate of intervention effect with a horizontal line extending either side of the block.

Forest Plot

Depending on the outcome in the studies, the CI can be (a) the relative likelihood of a given outcome for the intervention group vs the control group (e.g., live/die, success/failure) is measured by OR), or (b) the difference between treatment & control means (e.g. BP)

Forest Plot A vertical line is typically displayed to indicate no effect or null value, e.g. an OR = 1 or a mean difference of 0. Where ratios are presented (e.g. OR), null value is 1 and the scales are logarithmic. When the difference between means are presented, null value is zero (0) and the scales are normal.

Forest Plot

Effect of probio;cs on the risk of an;bio;c associated diarrhoea

Akobeng AK. Understanding systematic reviews and meta-analysis. Archives of disease in childhood. 2005;90(8):845-8.

Forest Plot

Understanding Risk vs Odds

Analyzing Dichotomous Outcomes

Risk vs Odds: Dichotomous Outcomes = when the outcome for every par@cipant is either one of two possibili@es or events; e.g., alive or dead, pain or no pain, pregnant or non-pregnant Expressing chance in terms of

RISK and ODDS

Risk vs Odds

Example: 12 people drank coffee

3 developed tachycardia

RISK of developing tachycardia?

Risk vs Odds

Risk:

3 developed tachy 12 who could have one

= = 25%

Risk = Number of events of interest Total number of observations

Risk vs Odds

Example: 12 people drank coffee

3 developed tachycardia

Odds of developing tachycardia?

Risk vs Odds

Odds (Total: 12)

3 developed tachy 9 no tachy

= 1/3 = 33.3%

Odds = Number of event of interest Number of observations WITHOUT the event

Risk vs Odds

Expressing in words Risk: the chance of tachycardia is one in four or 25%

Odds: the chance of tachycardia is one third (1/3) of the chance of no headache or

Odds: for every one person who develops tachycardia, three persons do not.

Do Risk and Odds Differ Much?

Scenario 1 N = 100; developed tachycardia: 5 Risk: 5/100 (5.00%), Odds: 5/95 (5.26%) Scenario 2 N = 100, developed tachycardia: 60 Risk: 60/100 (60%), Odds 60/40 (150%)

If number of observed events is BIG, risk and odds will differ a lot

If number of observed events is SMALL, risk and odds do not differ much

Example:

In patients with ACS (population), does the use of LMWH (treatment) decrease mortality and morbidity (outcomes) compared to UFH (control)?

Ref: Cohen M, Demers C, Gurfinkel EP, Turpie AG, Fromell GJ, Goodman S, Langer A, Califf RM, Fox KA, Premmereur J, Bigonzi F. A

comparison of low-molecular-weight heparin with unfractionated heparin for unstable coronary artery disease. Efficacy and Safety of

Subcutaneous Enoxaparin in Non-Q-Wave Coronary Events Study Group. N Engl J Med. 1997 Aug 14;337(7):447-52.

EFFICACY AND SAFETY OF SUBCUTANEOUS ENOXAPARIN IN

NONQ-WAVE CORONARY EVENTS (ESSENCE) STUDY GROUP 1997

Death at 48 hours

No death at 48 hours

Risk

Enoxaparin (n = 1607)

8 1599 0.005

UFH (n = 1564)

7 1557 0.004




RISK RATIO (RELATIVE RISK) = 0.005/0.004 = 1.13

Death at 48 hours

No death at 48 hours

Odds


8 1599 0.005

UFH (n = 1564)

7 1557 0.004




ODDS RATIO = 0.005/0.004 = 1.11

Risk Ra@o:

INTERVENTION RISK CONTROL RISK

Odds Ra@o:

INTERVENTION ODDS CONTROL ODDS

MI at 48 hours

No MI at 48 hours

Risk


11 1596 0.007

UFH (n = 1564)

14 1550 0.009




RISK RATIO (RELATIVE RISK) = 0.78

MI at 48 hours

No MI at 48 hours

Odds


11 1596 0.007

UFH (n = 1564)

14 1550 0.009




ODDS RATIO = 0.78

MI at 48 hours

No MI at 48 hours

Risk


11 1596 0.007

UFH (n = 1564)

14 1550 0.009




ABSOLUTE RISK REDUCTION: 0.009 0.007 = 0.002

Number Needed To Treat (NNT)

is the number of patients you need to treat to prevent one additional bad outcome (death, stroke, etc.).

For example, if a drug has an NNT of 5, it means you have to treat 5 people with the drug to prevent one additional bad outcome.

The ideal NNT is 1. The higher the NNT, the less effective is the

treatment


Calculation To calculate the NNT, you need to know the

Absolute Risk Reduction (ARR); the NNT is the inverse of the ARR:

NNT = 1/ARR

Where ARR = CER (Control Event Risk) - EER (Experimental Event Risk)

NNTs are always rounded up to the nearest whole number

MI at 48 hours

No MI at 48 hours

Risk


11 1596 0.007

UFH (n = 1564)

14 1550 0.009




NNT = 1/(ABSOLUTE RISK REDUCTION) = 1/0.002 = 5000


Example: The ARR is therefore the amount by which

your therapy reduces the risk of the bad outcome. For example, if your drug reduces the risk of a bad outcome from 50 per cent to 30 per cent, the ARR is: ARR = CER - EER = 0.5 - 0.3 = 0.2 (20%)

therefore NNT = 1/ARR = 1/0.2 = 5

Choosing a Statistical Test

Type of data

Quantitative analysis

Normal Distribution

T-test

>2 samples: One-way ANOVA

Skewed/kurtotic/not non-

parametric

Wilcoxon signed test/

Mann-Whitney

> 2 samples: Kruskal-Wallis

Qualitative analysis

Chi-square/Fisher exact (if

sample size small)

Extra Slides


Methods to test for heterogeneity: Q test: If Q test cannot reject the homogeneity

hypothesis ! use fixed-effects model because it is assumed that the estimated effect sizes only differ by sampling error

If reject homogeneity hypothesis ! use random-effects model that includes both within- and between-studies variability.


A problem with Q test is that it has poor power to detect true heterogeneity among studies when the meta-analysis includes a small number of studies and excessive power to detect negligible variability with a high number of studies


The other method is to use I2 index A meta-analysis with I2 = 0 means that all

variability in effect size estimates is due to sampling error within studies.

A meta-analysis with I2 = 50 means that half of the total variability among effect sizes is caused not by sampling error, but by true heterogeneity between studies.


If the confidence interval around I2 contains the 0% value, then homogeneity hypothesis is assumed.

Documents

Basic Concepts of Research