EXTERNAL QUALITY CONTROL OF MOLECULAR DIAGNOSTIC METHODS

Anton M. van Loon1,2,Colin Steel2, Calum Scott2 and Paul Wallace2

1 :Department of Virology, UMC Utrecht, The Netherlands2 :Quality Control for Molecular Diagnostics, Glasgow, Scotland

EU-QCCA

EXTERNAL QUALITY CONTROL ?

• Integral part of a laboratory quality management system

• Detection of weak spots in performance

• Improvement of reliability of results, gives confidence

• Comparison of performance with other laboratories

• Provision of uniform reference level when International Standards are lacking

• Expensive, but saves money

External Quality Control in Europe : 1998

• No uniform regulatory system for medical laboratories across Europe

• Most countries have QC organisations for external proficiency testing ( NEQAS, INSTAND, EQUALIS, SKMM etc.)

• Differences in approaches, but regular proficiency testing mainly focused on(virus)culture and serology.

• Lack of International Standards or reference reagents for molecular testing

• High rate of false-positives ( > 20 % ); sensitivity often unclear

• European Union Concerted Action for Quality Control (EU-QCCA)of MolecularMethods in Diagnostic Virology : Start in March 1998

• Endorsement by European Society for Clinical Virology and major diagnosticindustries (ILC)

European Union Concerted Action for Quality Controlof Molecular Diagnosis in Diagnostic Microbiology

(EU-QCCA)

AIM• To establish an external quality control programme for

assessment and evaluation of existing and evolvingnucleic acid amplification procedures in diagnostic virology

STRATEGY• Preparation and distribution of proficiency panels• Organisation of (virtual)post-distribution workshops• Production of reference materials e.g.‘run controls’

EU-QCCA PROFICIENCY PANELS

• Purpose : to help labs to determine their own performance• Composition determined by QCCA Working Groups• Consist of 8 - 12 samples• Include different sero/genotypes at various concentrations• Preferably inactivated and freeze-dried to improve stability

and facilitate shipment• Reporting time 4 - 6 weeks• Accompanied by questionnaire on technical details• Results of distribution published in detailed reports and

scientific publications

EU-QCCA Proficiency Panels : Number of Participants

Agent 1998/99 2000 2001

Enteroviruses 63 52 79

HSV 71 75 86

CMV - - 98

HBV 46 62 -

HCV 63 91 -

HIV 58 76 -

Chlamydia trachomatis

-

105 -

TB - - (80)

Enteroviruses

Viral Diagnosis• most frequent cause of aseptic meningitis• virus culture (CSF) increasingly replaced by NAT’s• NAT’S mostly in-house developed• sensitivity and specificity often unknown

Research• Association reported with chronic conditions as ALS, chronic

fatigue syndrome, post-polio syndrome

Enterovirus proficiency panels

• Panels consisting of 12 samples • Include various serotypes / genogroups (prototypes/

clinical isolates )• Include dilution series (PBS/ 5% FBS)• Consist of inactivated, freeze-dried culture supernatants• Production at UMC Utrecht; freeze drying at SVM• Predistribution testing at 2 reference laboratories• Shipment at ambient temperature, non-hazardous goods• Each panel accompanied by technical questionnaire

Distribution characteristics

1998/99 2000 2001

Participating laboratories 63 52 79 Number of countries 17 15 19

Reporting laboratories 59 46 68 Reported datasets 70 50 81 Evaluable datasets 69 47 78

FIRST EV PANEL: COMPOSITION AND RESULTS

Code Virus Titre

Reference labs

Participants (n=70)

TCID50/ml Geq/ml* 1 2 No positive

A01 EC9 104 107 + + 69

A02 EC9 102 104 + + 60

A03 Neg - - - - 2

A04 EC22 3.2x104 N.D. - - 3

A07 Neg - - - - 3

A08 PV2 1.6x103 104 + + 64

* : Based on experimental QPCR for CB3 (P. Muir)

FIRST EV PANEL: COMPOSITION AND RESULTS

Code Virus Titre

Reference labs

Participants (n=70)

TCID50/ml Geq/ml 1 2 No positive$

A05 CA9

4x102 107 + + 69

A12 CA9 4x102 107 + + 69

A06 CA9 40 106 + + 63 (62)

A09 CA9 4 105 + + 55 (54)

A10 CA9 4x10-1 104 - + 30 (28)

A11 CA9 4x10-2 103 - - 10 (8)

* : Based on experimental QPCR for CB3 (P. Muir) $ : Between brackets: No of correct dilution series

EV FIRST PANEL: QUALITATIVE ANALYSIS

Type of sample Code No of correct data sets (n=70)

Strong positive A01, 05, 06, 08, 12 59 (84%)

Positive A02, 09 52 (74%)

Weak-positive* A10, 11 31 (44%)

Negative A03, 07 66 (94%)

Strong positive, positive, negative

see above 46 (66%)

*: one or both samples /

1st EV PANEL: SCORE VS. TYPE OF ASSAY

99108-98-9Median

101324<7

1033137

3--3-8

2095249

182112310

7141111

TOTAL(n=68)

Roche(n=16)

Nested(n=26)

Semi-nested(n=11)

Single(n=15)

Type of PCRSCORE

PERFORMANCE SCORES ON QCMD EV-03

9.41--2811Other commercial

7.01--113NASBA

6.83--126Real-Time PCR

8.542171125Nested PCR

7.9312-511Semi-nested PCR

8.552-31020Single PCR

MEAN<778910- 11nASSAY

SCORE VS EXTRACTION METHOD : EV- A PANEL

Extraction

method N Score (points)

≤ 8 9 ≥ 10 Chaotropic agent 10 ΓΓ Φ Silica extraction 6 ΓΓΦ Γ Proteinase K 2 Γ Γ QIAmp Viral RNA 14 Γ ΓΦΦ ΓΓΦ Trizol reagent 5 ΦΦ TRI reagent 4 ΦΦ RNAzolB 3 Γ Γ High pure isolation 3 Γ Γ : single PCR Φ: semi-nested PCR : nested PCR

SCORE VS REVERSE TRANSCRIPTASE: EV- A PANEL

RT Enzyme

N Score (points)

≤ 8 9 ≥ 10

AMV 15 ΓΓΓΦΦΦ Φ ΓΦ

MMLV 25 ΓΦΦ ΓΓΓΦ ΓΓΦ

Γ : single PCR Φ: semi-nested PCR : nested PCR

SCORE VS DETECTION METHOD : EV - A PANEL

Detection Method

N Score (points)

≤ 8 9 ≥10

Gelelectro-phoresis

35 ΓΓΦΦΦΦΦ

ΦΦ ΓΦΦΦ

Hybridisation

16 ΓΓΓΓΓΦ ΓΓΓΓ ΓΓΓ

Γ : single PCR Φ: semi-nested PCR : nested PCR

QCMD Proficiency Testing : False-Positivity rate

1.0%--9Chlamydia trachomatis

2.4%--72Cytomegalovirus

-0.9%1.2%17 – 24Hepatitis C virus

-1.4%5.3%39 – 44Hepatitis B virus

-1.5%2.4%8 – 11Human immunodeficiency virus

5.8%8.1%4.8%89 – 95Herpes simplex virus

1.2%4.0%3.6%76 – 95Enteroviruses

200120001998/99% In-house assays

Agent

SCORE VS TYPE OF ASSAY: HIV

Panel 1 Panel 2

Assay No. Score No. Score

Commercial

8 7 ≤6 8 7 ≤6 bDNA 7 4 1 2 8 8 - - NASBA 7 0 4 3 8 1 2 5 RT-PCR 23

19 9

17 11 2

3 -

24 32

9 27

6 3

9 2

Hybrid Capture II - - - - 1 - - 1

In-house Qualitative 5 2 2 1 6 1 2 3 Quantitative 2 - 1 1 - - - -

Participant Results on First QCCA CytomegalovirusProficiency Panel- Qualitative ( n = 105 )

105

0

102

3

101

4

92

13

57

48

25

80

2

103 104

1

98

7

60

45

19

86

3

102

0

20

40

60

80

100

120

Number of

Data-Sets

80,000 20,000 5,000 1,300 320 80 0 5,000 1,300 320 80 0Viral Load (GEq/ml)

No. Positive No. Negative

QCMD - CMV01: Scores on Qualitative Results

ASSAY n 12-11 10 9 8 7 <7

Qualitative • in-house • commercial

57 14

9 3

9 1

19 4

12 5

6 1

2 0

Quantitative* • in-house • commercial

17 12

7 1

6 2

1 4

1 3

1 2

1 0

* : Quantitative results were converted to pos/neg results before scoring

PROBLEMS IN PROFICIENCY TESTING OF MOLECULAR DIAGNOSTIC ASSAYS: EXPERIENCE EU-QCCA

• Lack of international standard preparations

• Logistics : shipment / payments

• Inactivation/freeze drying (f.e. HBV, HCV)

• Panel samples to resemble clinical material (EV, HSV )

• Timely, high quality reporting to participants

• Achieving required level of quality/professionalism/QC system

• Costs

European Union Concerted Action for Quality Control of Nucleic Acid Amplification in Diagnostic Virology

STATUS (August 2001)

• Third year of program: EU sponsoring would stop bySeptember 2001; at the time, no suitable EU program identified for sponsoring (5th FP: QL, Growth; DG-SANCO)

• Request for continuation (ESCV, ESCMID, Industry)

• Need for further expansion of the program and increase in professionalism

• Business plan prepared, including grants from Industry

• QCMD : Quality Control for Molecular Diagnosis

QCMD STRATEGY

• Expand distributions : more pathogens and applications i.e. detection, quantitation, genotyping and sequencing, and more countries

• Develop International Standards and Working Reagents, partlyin collaboration with other organisations

• Set up an interactive web-site to present and promote the programme, to collect, publish and discuss results of distributions, and to provide a forum for technical and scientificdiscussions (www.qcmd.org)

• Present the programme and its results at scientific meetings, in scientific journals and to accreditation bodies etc.

REQUIREMENTS FOR AN EXTERNAL QUALITY ASSESMENTORGANISATION

- Independency : a not-for-profit organisation without commercial, financial or professional conflicting interests

- Transparancy : clear structure of organisation and of responsibilities

- Reliability : confidence in independent judgement

- Respectability : organisation managed by experts in the field

- Confidentiality : establishment of a neutral office

- Quality : quality management system, compliance with international guidelines/directives (ISO, CEN, EU)

UMCVirology

Board

EXECUTIVE

ILC (Industry LiaisonCommittee)

ESCMID

ESCV

Neutral &Executive Office

Contractors

TYPR.Schuurman

CNSA. Linde

STDJ. Schirm

BBVB. Niesters

RESPG. Ieven

QCMD Organisation Structure

OTHER

(WHO/EU)

EXECUTIVE : Graham Cleator, Paul Klapper,

Jim Reid, Anton M. van Loon (chair)

OFFICE : Paul Wallace

BOARD : Executive members

Chairs WP’s

Representatives ILC, ESCV, ESCMID,

major collaborating institutions

Quality Control for Molecular DiagnosticsWWW.QCMD.ORG

QUALITY CONTROL for MOLECULAR DIAGNOSTICS

Annual Program & Participant numbers

0

2

4

6

8

10

12

14

16

1998 1999 2000 2001 2002 2003

QCCA QCCA QCCA QCCA / QCMD QCMD

No.

of P

rogr

amm

es

0

200

400

600

800

1000

1200

1400

1600

No.

of P

artic

ipan

ts

No.programmesNo.participants

An International QC Programme

• Access to 107 countries worldwide

• 67 countries currently registered (general / interests)

• 45 countries actively participating in proficiency programmes

QUALITY CONTROL for MOLECULAR DIAGNOSTICS

QUALITY CONTROL for MOLECULAR DIAGNOSTICS

290

349

244

320

1 47

331 325

382

305

1 78

88

0

50

100

150

200

250

300

350

400

EV HSV VSV CMV EBV HIV HBV HCV CT TB NGo

P r ogr a mme I nt e r e st s

Ar ea Inter ests

QUALITY CONTROL for MOLECULAR DIAGNOSTICS

13 6

8 0

13 6

5 4

104 7

4

13

2 1

0

20

40

60

80

100

120

140

HIV MT TB MT HPV B19 JC SRS HAV Rota Res HHV6

Programme Interests (other)

Area Interests

Not final numbers *+Programme not startedxProgramme not done

+xxxxxLegionella pneumophila

+xxxxxNeisseria gonorrhoeae

91xxxxxVaricella zoster virus

+75xxxxEpstein Barr virus

+102xxxxHIV resistance genotyping

70*9698xxxCytomegalovirus

+8682xxxMycobacterium tuberculosis

122119x105xxChlamydia trachomatis

60*xxxxxHCV genotype

9283x8859xHepatitis C virus

7973x6145xHepatitis B virus

9069x7754xHuman immunodefiency virus

103102867368xHerpes simplex virus

10110577x5263Enterovirus

200320022001200019991998Programme

QUALITY CONTROL for MOLECULAR DIAGNOSTICS

The number of Participants for each distribution / year:

65.922.94.12.179.7984827411/05/2002MT

9.214.80.90.987.813081099807/03/2002HSV

90.014.60.00.684.8720906405/20/2002HIV

88.09.20.00.191.89921248005/20/2002HCV

51.511.92.10.189.2776977005/20/2002HBV

7.027.75.51.679.012001009306/10/2002EV

29.919.03.00.080.6603676112/17/2002EBV

87.946.00.60.966.8116011611103/18/2002CT

81.020.32.91.081.612601058911/05/2002CMV

%Commercial

assays

% False -veFrom totalpositiveSamples

% False +veFrom totalNegativeSamples

% equivocal

Totaltest

% correctTotal test

Total Tests

No. dataSets

No.Participants

Distribution Date

Programme Type

Summary of the 2002 Molecular QC programmes for Virology and Microbiology

QUALITY CONTROL for MOLECULAR DIAGNOSTICS

(University) Hospital

Laboratory61%

Reference Laboratory

6%

Public Health Laboratory

16%

Other2%

Manufacturer of Diagnostics

3%

Commercial / Private

Laboratory11%

STD Clinic Laboratory

1%

Research Laboratory

1%

QUALITY CONTROL for MOLECULAR DIAGNOSTICS

Major Participating Laboratory types:

Web30%

Fax62%

Letter8%

WebFaxLetter

Changes in Reporting types from 2002 to 2003

Web94%

Letter1%

Fax5%

QUALITY CONTROL for MOLECULAR DIAGNOSTICS

Summary and Conclusions

• An external QC organisation was established in 1998 (EU-QCCA) withpartial funding by EU

• Management by experts from the field• Fourteen distributions : 7 agents, 45-105 laboratories/distribution• Large variation in sensitivity, mainly with in-house methods• False-positivity rates between 1 - 8%; need for further improvement• EU-QCCA succeeded by QCMD (Sept. 2001)• Same approach with increase in quality/professionalism and dedicated,

full-time employees; compliance with guidelines (ISO, CEN, EU)• Commitment from ESCV, ESCMID and major diagnostic industry• Expansion of program to include more targets, applications, and also

more laboratories in more countries