Quality Assurance System_Ecotoxicity Studies_Breton et al 2009

A New Quality Assurance System for the Evaluation ofEcotoxicity Studies Submitted Under the New SubstancesNotification Regulations in CanadaRoger L Breton,*� Guy Gilron,` Ryan Thompson,§ Sara Rodney,� and Scott Teed�

�Intrinsik Environmental Science (formerly Cantox Environmental), 1550A Laperriere Avenue, Suite 103, Ottawa, Ontario K1Z 7T2, Canada`Teck, 3300-550 Burrard Street, Vancouver, British Columbia V6C 0B3, Canada§1333 Larose Avenue, Ottawa, Ontario K1Z 7X4, Canada

(Received 15 April 2008; Accepted 2 October 2008)

ABSTRACTNew substances destined for import into, or manufacture in, Canada must be reported to Environment Canada and Health

Canada under the New Substances Notification Regulations (Chemicals and Polymers) (NSNR). With the use of information

provided by the notifier, and other complementary information available to the 2 departments, the New Substances Program

conducts ecological and human health risk assessments. Over the past 10 y, more than 750 ecotoxicity studies have been

submitted to the New Substances Program of Environment Canada under the NSNR. Most of these experimental studies are

not publicly available but are useful in the ecological risk assessment of new substances and for the development of

Quantitative Structure–Activity Relationships (QSARs). In this paper, we describe the development and validation of a

computer-based scoring system and our approach in the development of scoring methods used to assess the quality and

usability of ecotoxicity studies with fish, Daphnia spp., and green algae. Results of ranking exercises conducted with these

methods are described and discussed, together with the potential use of these results in a regulatory context. In addition,

the methods are discussed in comparison with other similar evaluation schemes described in the literature.

Keywords: Ecotoxicity tests Quality assurance New Substances Program Ecological risk assessment

OECD Test Guidelines

INTRODUCTIONNew substances imported into, or manufactured in, Canada

must be reported to Environment Canada under the NewSubstances Notification Regulations (Chemicals and Poly-mers) (NSNR) of the Canadian Environmental ProtectionAct, 1999 (Canadian Parliament 1999). The NSNR wereinitially promulgated on 1 July 1994 (GC 1994) and wererecently amended on 31 October 2005 (GC 2005). Uponreceipt of these notifications, Environment Canada andHealth Canada conduct ecological risk assessments (ERAs)and human health risk assessments, respectively, on the basisof information supplied by the notifier and other informationavailable to the departments.

The New Substances Program (NSP) of EnvironmentCanada receives approximately 800 new substance notifica-tions (NSNs) per year. Since its inception in 1994, nearly16000 NSNs have been received. On the basis of specificcriteria, a number of these NSNs are accompanied byexperimental ecotoxicity studies as part of the requirementsof the NSNR. To date, the NSP has received more than 750ecotoxicity studies.

Ecotoxicity studies are usually conducted according tonationally or internationally approved standard methods.However, the quality of the data from these studies can vary,depending on a number of factors, including how rigorouslyexperimental procedures were followed by the contractlaboratories, the intrinsic properties of the substance under

investigation (e.g., water solubility, vapor pressure), andwhether the studies were conducted in accordance with goodlaboratory practice (GLP) principles (OECD 1998). Evalua-tion specialists in the NSP must evaluate the quality of theecotoxicity studies and determine whether to accept or rejectthe studies and notifications. To support this task, the NSP ofEnvironment Canada commissioned the development of aquality assurance system for the evaluation of ecotoxicitystudies for the 3 major toxicity tests required—specifically,fish, Daphnia spp., and green algae.

The quality assurance system was integrated into a multi-functional electronic program using a Microsoft Accessformat. The electronic scoring methodology program is calledthe Ecotoxicological Quality Evaluation System and ScoringTool (eco-QESST). The eco-QESST program has a number ofbenefits; specifically, it builds program consistency in evalu-ating the reliability of ecotoxicity studies for use in ERAs; itassists in evaluating and comparing the quality and robustnessof ecotoxicity studies received by Environment Canada insupport of the NSNRs; it provides transparency in decision-making in new substance risk assessment; it provides adetailed electronic record of the studies, allowing for accessto key study parameters; it provides ready access to surrogatedata for the assessment of analogous substances; and itidentifies studies that could be useful for quantitativestructure–activity relationship (QSAR) model development.

Since 1994, approximately 80% of the ecotoxicity teststudies submitted to Environment Canada were conductedaccording to, or consistent with, Organization for EconomicCooperation and Development (OECD) Test Guidelines(Figure 1). As such, eco-QESST modules were developedfor fish, Daphnia spp., and green algae on the basis of OECD

* To whom correspondence may be addressed:[email protected]

Published on the Web 10/7/2008.

Integrated Environmental Assessment and Management — Volume 5, Number 1—pp. 127–137� 2009 SETAC 127

Orig

inalRese

arch

Review

Test Guidelines 203 (OECD 1992), 202 (OECD 2004), and201 (OECD 2002), respectively. The calculated toxicityendpoints of interest are: The concentration estimated to belethal to 50% of the fish in a 96-h test exposure (i.e., 96-hLC50), the concentration estimated to cause an effect (i.e.,immobility or mortality) to 50% of daphnids in a 48-h testexposure (i.e., 48-h EC50 or LC50), and the concentration oftest substance that results in a 50% reduction in algal growthor growth rate relative to the control in a 72-h test exposure(i.e., 72-h EC50/IC50).

A systematic approach for evaluating the quality, andtherefore the reliability, of ecotoxicity studies had beenproposed by Klimisch et al. (1997) for use in the InternationalUniform Chemical Information Database. The Klimisch et al.(1997) approach was intended to harmonize data evaluationprocesses worldwide and served as a good source fordeveloping the eco-QESST system. Quality assurance criteriareceived from the US Environmental Protection Agency,Office of Prevention, Pesticides and Toxic Substances, werealso considered for this purpose. Moreover, Hobbs et al.(2005) recently published a general evaluation scheme forecotoxicity studies, with many similarities between it and theeco-QESST system. A comparison of the 2 approaches ispresented below in the Results and Discussion section.

To date, 614 OECD studies have been evaluated (i.e., 270fish, 213 Daphnia spp., and 131 green algae studies) with eco-QESST modules developed for these methods. In this paper,we describe the development of the eco-QESST system andthe results obtained in the evaluation of the 614 studies, andwe compare it to the structure and application of anothersimilar scheme used elsewhere (Hobbs et al. 2005).

METHODSThe eco-QESST system was initially developed by carefully

reviewing the OECD Test Guidelines and identifying the keyelements related to study quality. Questions were developedto evaluate the quality of ecotoxicity test studies with the

objective of determining whether key elements of these testsare provided in standard ecotoxicity test reports. Thesequestions were presented in issue papers and discussed inworkshops held in conjunction with Environment Canada andHealth Canada staff. Recommendations were made on therelative importance (i.e., weighting of questions), as well asthe relevance of the questions and elements identified, on thebasis of NSP requirements.

The eco-QESST system was first developed for fish,followed by Daphnia spp., and finally green algae ecotoxicitytests. An iterative process was used to improve consistencyamong the 3 modules. More specifically, questions andelements (e.g., physicochemical properties of the testsubstance, use of controls, and appropriate use of statistics)common to all species were included in each module. Thishelps to build consistency in the scoring systems and allowsfor direct comparisons among the modules. Additionalrefinement to the modules for fish, Daphnia spp., and greenalgae is likely to occur as studies continue to be reviewed andas experience with the system is gained by NSP evaluationspecialists.

The eco-QESST system is a multifunctional electronicprogram featuring storage of study data, while providing asimultaneous evaluation of study quality. The system allowsfor the capture of information on the notifier and reportedsubstance, the ecotoxicity data (and related calculatedstatistical endpoints), elements relating directly to studyquality, and technical information that reflects study quality(e.g., physicochemical data, test conditions). The output ofthe system also provides a variety of reporting formats thatallow for flexibility and provides valuable information neededto support the acceptance or rejection of an ecotoxicity study,important for use, including the preparation of ERAs.

To facilitate the use of eco-QESST, an interactive interfaceprompts the user to address elements related to study qualityin a question-and-answer format. Each question is supportedby a help system that provides guidance on the various

Figure 1. Relative percentage of studies conducted with the use of ecotoxicity test methods for fish, Daphnia spp., and green algae under the New SubstancesProgram (1984–2004).

128 Integr Environ Assess Manag 5, 2009—RL Breton et al.

elements that should be reported in the test study. Moreover,answers to the questions are set up in a drop-down menuformat. This allows for consistency in choosing the appro-priate answer and enables automation of the calculation of anoverall study quality score.

The questions used to evaluate the quality of ecotoxicitytests for fish, Daphnia spp., and green algae, on the basis ofOECD Test Guidelines, are presented in Table 1. Alsoprovided in this table are the weights (i.e., on that basis ofrelative importance) attributed to each answer. The questionsare grouped into 5 categories: 1) test method, 2) test species,3) test substance, 4) test design and conditions, and 5) validityof test.

In the eco-QESST system, the majority of questions areanswered as either ‘‘Yes,’’ ‘‘No,’’ ‘‘NA’’ (not applicable), or‘‘NR’’ (not reported). A ‘‘Yes’’ answer is given a fixed weightdepending on the relative importance of the quality elementaddressed by the question. A ‘‘No’’ answer is always allocateda weight of zero. ‘‘NA,’’ or not applicable, refers toinformation that is not relevant to the study under inves-tigation and is therefore excluded altogether from thecalculation of the overall study quality score. For example,the question pertaining to the appropriate use of a solventcarrier (and solvent control) in a study would be answered‘‘NA’’ if the study was conducted with a test substance that isstable, has low volatility, and is water soluble. This is animportant feature that allows for flexibility in reviewingquality elements that are not relevant for every study. ‘‘NR’’ isused for cases in which a quality element is applicable to thestudy, but the information is not provided in the laboratorytest report. An ‘‘NR’’ answer is assigned a weight of zero andtherefore decreases the overall study quality score. Inevitably,a few quality elements could not be captured by these basic 4answers; therefore, multiple-choice answers are used. Theelements captured with the basic answers and multiple-choiceanswers are presented in Table 1. A help system that providesguidance on elements that should be reported in the teststudy was developed; however, it is beyond the scope of thispaper.

The overall study quality score (OSQS) is calculated as apercentage of total as follows:

OSQS ¼

Xn

i¼1

½ðWYÞi þ ðWMCÞi�

Xn

i¼1

½ðWYÞi þ ðWNÞi þ ðWNRÞi þ ðWMCmaxÞi�

where WY¼weight of question when ‘‘Yes’’ is the answer forquestion i; WMC ¼ weight of answer selected in multiplechoice for question i; WN¼weight of question when ‘‘No’’ isthe answer for question i; WNR ¼ weight of question whenNR is the answer for question i; and WMCmax ¼ maximumweight of multiple choice question for question i.

In essence, the above equation refers to the sum of theweights obtained when answering all the questions, dividedby the maximum weight that could have been obtained. Theoverall study quality score is used to assign the study to aquality or reliability level. Conversion criteria are presented inTable 2.

The qualitative descriptive levels of quality (e.g., ‘‘verygood’’) were chosen in consultation with workshop partic-ipants using professional judgment and can be modified tosuit the user’s needs, if applied in other frameworks or for

other jurisdictions. Table 2 provides the current descriptivelevels of quality used by evaluation specialists within the NSP.It is important to note that the overall study quality score isnot intended to be used as the only determining factor bywhich an ecotoxicity study is evaluated (i.e., it is not a pass/fail system). Expert judgment is still required to addresswhether any missing or unsatisfactory quality elements aresufficient to completely invalidate a study. To accommodateexpert judgment, ‘‘intelligence’’ has been built into the eco-QESST system by alerting the user’s attention to criticalquality elements by way of flagging them, such as, forexample, meeting test validity criteria and evaluation of thescientific integrity of the study. These flags were identifiedduring the workshops. Currently, evaluation specialists usingthe system are prompted to provide a decision with properrationale for every flag raised during the review of a givenstudy.

The robustness of the eco-QESST system was tested byevaluating the quality of 614 ecotoxicity studies (i.e., 270fish, 213 Daphnia spp., and 131 green algae studies). Thevariability of scores obtained by different eco-QESST userswas evaluated by conducting a 2nd round of reviews of 20% ofthe 614 studies. The studies were selected at random, anddifferent reviewers were assigned to rescoring the studies. Fivereviewers took part in this rescoring validation exercise, andthey had a combined 47 y of experience evaluatingecotoxicity studies used for guideline development andecological risk assessment. One of these reviewers also servedfor more than 10 y as a quality assurance officer and studydirector for numerous GLP studies (analytical, ecotoxicity,and mammalian toxicology) for chemical and polymerproducts and pesticides. This validation exercise generateddiscussion on the interpretation of guidance associated witheach question. As a result of this exercise, the guidanceprovided to users of the eco-QESST system was iterativelyrefined to provide more consistent evaluation of the studies.

A comparison of the maximum scores for each of the 5categories of questions was conducted to determine theconsistency among the 3 eco-QESST modules. The maximumscores of each quality element from a given category wereadded to obtain the total score for that category. Ensuring abalanced score for each category among eco-QESST modulesshould yield a more robust and comparable evaluation systemand provide equitability when comparing the quality andreliability of total overall study quality scores. The categoryscores for all 3 modules are presented in Table 3.

RESULTS AND DISCUSSIONTo our knowledge, the only published evaluation frame-

work that is similar in scope and design to the eco-QESST isthe recently published scheme reported in Hobbs et al.(2005) that is based on original questions proposed byMarkich et al. (2002), who used the scheme to evaluatestudies in the Australasian Ecotoxicity Database (AED).Although we discussed the Klimisch et al. (1997) schemeearlier, we have only compared the eco-QESST system to theHobbs et al. (2005) scheme because, unlike the Klimisch et al.(1997) scheme, only the eco-QESST and the Hobbs et al.(2005) schemes use formal numeric scores that integrateindividual scores from individual evaluation elements.

A comparative assessment of the AED evaluation scheme(as evaluated and refined) and the eco-QESST system(reported in this study), is provided in Table 4.

System for the Evaluation of Ecotoxicity Studies—Integr Environ Assess Manag 5, 2009 129

Table 1. Questions, relative weights, and question types for evaluating studies of fish, Daphnia spp., and algae

Question/Weight/Question typea

Does the question apply?

Fish Daphnia Algae

Was the study conducted according to a recognized international standard? Yes Yes Yes

Weight:

International Standard Method (3)

Justified Comparable Guideline (3)

Unjustified Comparable Guideline (2)

Unrecognized method (0)

Not reported (0)

Question type: Test method

Was the study consistent with OECD Good Laboratory Practice (GLP) principles? Yes Yes Yes

Weight: 3

Question type: Test method

Did the study conduct range-finding and definitive tests? Yes Yes Yes

Weight: 1

Question type: Test design and conditions

Was the scientific name (Latin) reported? Yes Yes Yes

Weight: 1

Question type: Test species

Was the strain reported? No No Yes

Weight: 1

Question Type: Test Species

Was the life cycle stage of test species/age reported (days)? Yes No No

Weight: 1


Was the age of the daphnids appropriate? No Yes No

Weight: 1


Was the age of the culture prior to inoculation appropriate? No No Yes

Weightb:

Yes full (2)

Yes partial (1)

Not reported (0)


Was the initial cell concentration sufficient for running the test? No No Yes

Weight: 1


Was the length of the species appropriate (cm)? Yes No No

Weight: 1


Was the mean weight of the species reported (g)? Yes No No

Weight: 1


Table 1. Continued



Fish Daphnia Algae


Were the test organism acclimation conditions and/or feeding periods prior to testingappropriate?

Yes No No

Weight: 1


Was the method and medium of cultivation appropriate? No No Yes

Weightc:

Yes full (2)

Yes partial (1)

Not reported (0)


Was the purity or composition of tested substance reported? Yes Yes Yes

Weight: 3

Question type: Test substance

Were the water quality characteristics of the dilution water reported? Yes Yes No

Weight: 1


Was the test solution preparation appropriate? No No Yes

Weight: 1


Was the measured water solubility of the test chemical reported (mg/L)? Yes Yes Yes

Weight: 1

Question type: Test substance

What was the test system design? Yes Yes No

Weight:

Flow-through (3)

Static–renewal (2)

Static (1)

Not reported (0)

Question Type: Test Design and Conditions

Was a loading density sufficient for running the test? Yes(e.g.,�1.0 g

/L)

Yes (e.g., atleast 2 mLfor eachanimal)

No

Weight: 1


Was the number of organisms reported for each test solution? Yes Yes No

Weight: 1


Was the number of replicates (including controls) reported? Yes Yes Yes

Weight: 1


Table 1. Continued



Fish Daphnia Algae


Was the exposure period reported (hours/days)? Yes Yes Yes

Weight: 1


Did pH remain in the range of the standard method? Yes(OECD:6–9)

Yes (OECD:6–8.5)

No

Weight: 3


Was the temperature within appropriate range for test species or range of standardmethod?

Yesd Yes (e.g.,18–22 8C)

Yes(e.g.,21–26

8C)

Weight: 1


Was the dissolved oxygen concentration at least 60% of the air saturation valuethroughout the test?

Yes Yes No

Weight: 2


Was the daily photoperiod appropriate (hours)? Yes Yes No

Weight: 1


Was the test conducted under continuous light? No No Yes

Weight: 1


Was the light intensity appropriate? No No Yes

Weight: 1


Was a carrier control used? Yes Yes Yes

Weight: 1


Were the test endpoints measured/observed at the intervals required by the test meth-od (e.g., every 24 hours)?

Yes Yes No

Weight: 1


Were the observations of growth monitored in the controls at the intervals requiredby the test method (e.g., every 24 hours)?

No No Yes

Weight: 1


Were the appropriate statistical procedures used, described, and appropriately re-ported?

Yes Yes Yes

Weight: 2

Question type: Validity of test


Table 1. Continued



Fish Daphnia Algae

Was the mortality/immobilization in the control(s) reported at the end of the test? Yes Yes No

Weight:

�10% (3)

.10 to �20% (1)

.20% (0)


Did the number of algal cells in the controls increase by a factor of more than 16 in72 hours?

No No Yes

Weight: 1


Was temperature measured and maintained? Yes(61 8C)

Yes (61 8C) Yes (628C)

Weight: 2


Was pH measured and maintained? Yes(61.0units)

Yes (61.5units)

Yes(61.5units)

Weight: 2


Was DO measured and maintained (620%)? Yes Yes No

Weight: 2


Were the test substance concentrations measured and maintained? Yes Yes Yes

Weight: 3


Did the coefficient of variation for daily growth rates in control cultures during thecourse of the test (days 0–1, 1–2, 2–3) exceed 35%?

No No Yes

Weight: 2


Did the coefficient of variation for average growth in replicate control cultures exceed 15%? No No Yes

Weight: 2

Question type: Validity of testa OECD¼ Organization for Economic Cooperation and Development; DO ¼ dissolved oxygen.b Algal cultures used in tests are required to be of a certain ‘‘age’’ (e.g., 2–3 d before inoculation), demonstrated to be in the exponentialgrowth phase, or both. This should be confirmed (e.g., evidence that growth of the culture was monitored and graphed to confirm that itwas in log growth. The curve follows a ‘‘typical’’ sigmoidally shaped exponential growth curve (this is usually determined by eye, and doesnot need to be quantified). To obtain a full score for this question, the study must report a culture age before inoculation of between 2and 7 d and be graphed. If only 1 of these 2 data is available, then a partial mark is given to this question. Otherwise, answer ‘‘no’’ if thedata are not appropriate, or answer ‘‘not reported’’ if the data are not available in the study report.

c To get a full score, culture conditions and composition of the culture medium need to be provided in the study report. The conditions used(e.g., culturing apparatus, incubation period, temperature, light intensity) to initiate and culture the test organisms (e.g., inoculation ofcells in aqueous medium from solid-phase agar medium), the composition of the medium (i.e., chemical or elemental composition orboth), including water type (e.g., distilled, reconstituted, etc.), should be reported. A partial score is given if information for either cultureconditions or medium is reported in the study. Otherwise, answer ‘‘no’’ if the data are not appropriate, or answer ‘‘not reported’’ if thedata are not available in the study report.

d Temperature ranges recommended for fish species according to OECD Method 203 (OECD 1992) is 21 to 258C for Brachydanio rerio,Pimephales promelas, Oryzias latipes, Poecilia reticulata, and Lepomis macrochirus; 20 to 248C for Cyprinus carpio; and 13 to 178C forOncorhynchus mykiss.


Despite some relatively minor differences in scope (e.g.,substances being evaluated, nature and format of the studiesevaluated), the 2 schemes (Table 4) share a number offundamental similarities, the most important of which are asfollows:

� The schemes are both used to evaluate the quality andrigour of aquatic toxicity studies for use in ERAs;� Weighted answers to questions are used to develop a

quality or overall score (however calculated differently)to classify ecotoxicity studies;� The general nature of the questions used to evaluate

studies;� The use of general qualitative classification categories

(i.e., unacceptable, acceptable, high quality), on the basisof the final scores; and� The overall basis for evaluation and improvement of the

evaluation schemes.

The similarities that emerged from this comparison supportmany of the aspects incorporated into the development of theeco-QESST system.

The distributions of overall study quality scores for the270 fish, 213 Daphnia spp., and 131 green algae studies werevery similar (Figure 2). Ten percent of the fish studies

obtained a score of more than 90% (i.e., ‘‘very good’’),whereas 75% of the studies had a score greater than 70%(i.e., ‘‘good’’ and ‘‘very good’’ combined). This implies that65% of the studies were, overall, of good quality. Thirteenpercent of the fish studies had scores of acceptable only (i.e.,60%–70%). Similarly, 10% and 5% of the Daphnia spp. andgreen algae studies, respectively, scored above 90% (i.e.,‘‘very good’’), whereas 81% and 66% of the Daphnia spp.and green algae studies, respectively, scored above 70% (i.e.,‘‘very good’’ and ‘‘good’’ combined). Only 9% of theDaphnia spp. studies obtained scores classified as ‘‘accept-able’’ (i.e., 60%–70%). The biggest difference with the greenalgae studies was that a greater number of them (i.e., 21%)had acceptable scores only, and no studies obtained a scoreof less than 40%. For all 3 test guidelines, 87% or more of thestudies had at least an ‘‘acceptable’’ rating (i.e., a score.60%) according to the eco-QESST system developed forOECD Test Guidelines.

The consistency of the eco-QESST system among differentusers was validated by rescoring 20% of the 614 studies,

Table 3. Maximum scores for each category of questions forfish, Daphnia spp. and green algae

Category of question

Maximum score

FishDaphnia

spp.Greenalgae

Test method 6 6 6

Test species 4 2 4

Test substance 4 5 4

Test design and conditions 22 20 15

Validity of test 11 11 11

Total 47 44 40

Table 2. Study quality levels based on overall score ineco-QESSTa

Overall study quality score (%) Quality level

0–60 Not reliable

.60–70 Acceptable

.70–90 Good

90–100 Very gooda eco-QESST ¼ Ecotoxicological Quality Evaluation System andScoring Tool.

Figure 2. Distribution of overall study quality scores for fish, Daphnia spp., and green algae studies.


which led to the refinement of the help system and guidanceto evaluators; moreover, different users often producedidentical scores for the same study. For cases in whichdiscrepancies occurred, scores obtained were typically within65% overall study quality score, with the exception of 1rescoring study for fish, Daphnia spp., and algae (i.e., 5.02%for fish, 6.98% for Daphnia, and 5.71% for algae). Figures 3a,b, and c present the percent difference in scoring between theevaluators for the 20% rescoring validation exercise. The65% discrepancy could be attributed to the following factors.

� Differences in interpretation of the information providedin the test report.

� Ambiguity of the information provided in the testreports.

� Information that was overlooked by the user on reviewingthe test report (e.g., information provided in an appendixor footnote).

A comparison of the maximum scores for the 5 categoriesof questions was conducted to determine consistency amongthe 3 eco-QESST modules. The results indicated that themaximum scores for each of the categories are either identicalor very similar across the 3 modules (Table 3). The onlyexception was the ‘‘test design and conditions’’ category forgreen algae. This category yielded a maximum score of 12compared with 19 and 17 for fish and Daphnia spp. modules,respectively. The reason for this was that the OECD TestGuidelines for both fish and Daphnia spp. require more keyelements in this category to meet testing specifications.

As illustrated in Figure 2, the overall study quality scoresobtained with the eco-QESST system can be characterized asa lightly right–skewed normal distribution. To corroboratethis observation, a qualitative evaluation of the distribution ofoverall study quality scores from eco-QESST was conductedby experienced evaluation specialists from the NSP. On thebasis of their previous experience in reviewing ecotoxicitystudies for acceptability and use in ERAs (prior to the use ofthis system), the evaluation specialists believed that, overall,these distributions were quite representative of the quality ofthe studies submitted and evaluated in support of NSNs.

The eco-QESST system has 2 other important functions: Itcan be used as a searchable database for identifying surrogatedata and as a screening tool to conduct a preliminary searchfor studies useful in QSAR model development. The latter isvery important in supporting ERA activities within the NSPfocused on scientific weight-of-evidence approaches. How-ever, improvements to the eco-QESST system will berequired because it was observed that studies consideredappropriate for QSAR model development are not necessarilythose with the highest scores. Appropriate dose–responsecurves are essential for QSAR development and not alwaysobserved in high-quality studies that report ‘‘limit tests’’ (i.e.,tests that only use a negative control and 1 exposureconcentration, with no concentration gradient).

It is imperative to ERA activities of the NSP that notifierssubmit ecotoxicity data and studies that are of high quality.High-quality studies help to reduce uncertainty in ERAs, andultimately in decisions taken by the NSP. The informationpresented in this paper demonstrates that the majority ofecotoxicity studies provided to the NSN program have beenof acceptable quality for conducting ERAs.

Nonetheless, many studies scored in the ‘‘acceptable’’ and‘‘not reliable’’ ranges. Evaluation specialists with the NSP

believe that data routinely generated when conducting

ecotoxicity studies are sometimes not included in study

reports because laboratories and notifiers might not be aware

that this information is required for evaluating the quality and

reliability of these studies. This paper serves as an important

reference to stakeholders and will, one hopes, improve the

awareness of the process and communication among notifiers,

contract laboratories, and the NSP of Environment Canada.

Improved communication will ensure that all pertinent

information is reported and available to Environment Canada

evaluation specialists. This is especially important for diffi-

cult-to-test substances, for which details related to study

design are crucial in evaluating the integrity of the results.

CONCLUSIONSThe eco-QESST system is a highly effective decision tool used

for evaluating the quality and reliability of fish, Daphnia spp.,

and green algae ecotoxicity testing studies on the basis of OECD

Test Guidelines. The system provides a standardized review

Figure 3. (a) Percent differences in rescoring validation between evaluatorsfor fish ecotoxicity studies. (b) Percent differences in rescoring validationbetween evaluators for Daphnia spp. ecotoxicity studies. (c) Percent differ-ences in rescoring validation between evaluators for green algae ecotoxicitystudies.


Table 4. Comparison of 2 quality evaluation schemes for review of ecotoxicity studiesa

Aspect AED schemeb eco-QESST

Scope and objective of evaluationscheme

Derivation of water quality guidelines(Australia and New Zealand) fromUSEPA ECOTOX database and AEC

Regulatory ERA of new substances;preliminary evaluation of data foruse in QSAR development for newsubstance ERAs (Canada)

Basis for evaluation Quality and scientific rigor of studiesfor use in hazard and risk assess-ments in the derivation of waterquality guidelines

Quality and rigor of studies for ERAsby regulatory evaluators

Evaluation specialists 55 ecotoxicological scientists (fromgovernmental and nongovernmentalorganizations): postgraduate stu-dents and scientists with between 2and 25 y of experience

Evaluators from the New SubstancesBranch of Environment Canadawith between 3 and 20 y of experi-ence evaluating ecotoxicity studies

Substances evaluated Metals New substances; mainly organic sub-stances

Basis for methodology Weighted answers to questions result-ing in a quality score (see below)

Weighted answers to questions result-ing in an overall quality score

Nature of questions Overall, questions are similar in scopeand detail

Overall, questions are similar in scopeand detail

Weighting of questions Generally 3 categories: Generally 3, 2, 1, and 0

10 or 0 (assume Major type)

4/5 or 0 (assume Intermediate type)

3 or 1 or 0 (assume Minor type)

Existence of guidance for questions Not reported; assume that there is noquestion guidance for evaluators

Reported, but not provided in thispublication

Nature of studies Primary literature studies, aquatic toxi-city

Chemical substance studies (generallynot published in the primary litera-ture), some of which are conductedaccording to GLP principles

Number of studies evaluated 2 (from the primary literature); evalu-ated by 55 scientists

614 (from ecotoxicity study reportssubmitted under the NSNR); evalu-ated by 5 scientists

Scoring endpoint Quality score ¼ total score/total possi-ble score

See the equation in the text

Qualitative quality score classifications Three categories Four categories

,50%, Unacceptable 0–60%, Not reliable

51–79%, Acceptable .60%–70%, Acceptable

.80%, High quality .70%–90%, Good

90%–100%, Very good

Basis for improvement Interevaluator variability Interevaluator variability to be im-proved by guidance

Approach for refinement and revisionof scoring scheme

Evaluation Consensus-building workshop

Survey Evaluation

Re-evaluation Consensus-building workshop/re-evaluation and refinement

Refinement Consensus between scoring meth-odologies for EC method–basedversion of eco-QESST


process that can be applied to ecotoxicity studies submitted tothe NSP. In so doing, it reduces individual variation amongevaluation specialists when assessing studies. Therefore, eco-QESST is a transparent system providing targeted questions, astandardized scoring algorithm, assigned weights to questions,and guidance that identifies key elements needed to answer thequestions.

Although most ecotoxicity studies provided as part ofNSNs to Environment Canada were classified as ‘‘good’’studies, it is believed that the quality of these studies, andmore importantly, those which scored below 70% (i.e.,‘‘acceptable’’ and ‘‘not reliable’’), could be improved. More-over, it is believed that relatively few additional data wouldneed to be reported in most study reports to elevate themfrom ‘‘good’’ to ‘‘very good’’ quality. The data necessary toanswer all questions are likely routinely generated duringtesting, but might be absent from ecotoxicity test reportsbecause they are either not required to be reported or thenotifiers (and their contract laboratories) could be unaware ofthe usefulness of these data for evaluating the quality ofstudies and, in turn, for conducting new substance ERAs. Thispaper serves as a useful reference to notifiers and laboratoriesof the questions used to evaluate the quality and scientificintegrity of ecotoxicity studies. Providing the data necessaryto address the eco-QESST system questions reduces un-certainty in ecotoxicity study data submitted as part of NSNsand increases confidence in the results of new substanceERAs. The eco-QESST system also serves as a usefulcommunication tool for evaluation specialists, notifiers,contract laboratories, or a combination of participantsconducting and evaluating ecotoxicity tests.

Acknowledgment—Funding was provided by the New Sub-stances Program of Environment Canada. Special thanks toMichel Lortie of Environment Canada for his continuedparticipation, support, and enthusiasm for publishing thismanuscript. Andy Atkinson, Adriana Glos, and Mark Lewis, allof Environment Canada’s New Substances Division, provided

critical input to this paper. Rick Scroggins and Lisa Taylor from

Environment Canada’s Method Development and Applica-

tions Section also provided valuable input to the process. We

are also grateful to Dale Chambers of Normal Data for the

software development of the eco-QESST. We also acknowl-

edge the late Vince Nabholz (US Environmental Protection

Agency, Office of Prevention, Pesticides and Toxic Substances)

for providing us with information on quality assurance criteria

for assessing the quality of ecotoxicity studies.

REFERENCESCanadian Parliament. 1999. Canadian Environmental Protection Act. Can Gaz

22:38–76.

[GC] Government of Canada. 1994. New substances notification regulations. SOR/

1994-260 Part II. Can Gaz 128:1443–1495.

[GC] Government of Canada. 2005. New substances notification regulations

(chemicals and polymers). SOR/2005–247 Part II. Can Gaz 139:1864–1928.

Hobbs DA, Warne MSt-J, Markich SJ. 2005. Evaluation of criteria used to assess

the quality of aquatic toxicity data. Integr Environ Assess Manag 1:174–

180.

Klimisch H-J, Andreae M, Tillmann U. 1997. A systematic approach for evaluating

the quality of experimental toxicological and ecotoxicological data. Regul

Toxicol Pharmacol 25:1–5.

Markich SJ, Warne MSt-J, Westbury AM, Roberts CJ. 2002. A compilation of data

on the toxicity of chemicals to species in Australasia. Part 3: Metals. Australas J

Ecotoxicol 8:1–72.

[OECD] Organization for Economic Cooperation and Development. 1992.

Guideline for testing of chemicals—Fish, acute toxicity test (Method 203).

Paris (FR): OECD.

[OECD] Organization for Economic Cooperation and Development. 1998. OECD

principles of Good Laboratory Practice (as revised in 1997). Environment

Directorate. Paris (FR): OECD.


Guideline for testing of chemicals—Draft revised Guideline 201: Freshwater

alga and cyanobacteria, growth and inhibition test. Paris (FR): OECD.


Guideline for testing of chemicals. Daphnia sp., acute immobilization test

and reproduction Test Method 202. Paris (FR): OECD.

Table 4. Continued

Aspect AED schemeb eco-QESST

Reasons for interevaluator variability Evaluators have different interpreta-tions of the information providedin study

Evaluators have different interpreta-tions of the information providedin study

Evaluators might fail to find infor-mation in study

Evaluators might fail to find infor-mation in study

Evaluators might have different in-terpretations of questions andscoring scheme

Evaluators might have different in-terpretations of questions andscoring scheme

a AEC ¼ Australasian Ecotoxicity Database; AED ¼ Australasian Ecotoxicity Database; EC ¼ European Commission; eco-QESST ¼Ecotoxicological Quality Evaluation System and Scoring Tool; ERA ¼ ecological risk assessment; GLP ¼ good laboratory practice; NSNR ¼New Substances Notification Regulations; QSAR¼quantitative structure–activity relationship; USEPA¼US Environmental Protection Agency.

b Markich et al. (2002) and Hobbs et al. (2005).


Documents

Quality Assurance System_Ecotoxicity Studies_Breton et al 2009