Upload
others
View
0
Download
0
Embed Size (px)
Citation preview
EINFÜHRUNGSVERANSTALTUNG CSP 2018 QUEST-KRITERIEN
Berliner Institut für Gesundheitsforschung13. September 2018
Die folgenden Folien sind als Anregung zu verstehen,
sie ersetzen nicht die AKTIVE Auseinandersetzung mit den
QUEST-Kriterien im Hinblick auf Ihre spezifische
Forschungsfrage.Bei Fragen wenden Sie sich gerne an Dr. Miriam Kip ([email protected] oder
QUEST Tool box: (https://www.bihealth.org/en/quest-center/mission-approaches/englische-
uebersetzung/the-quest-toolbox/)
mailto:[email protected]
MERIT-Project
Miriam Kip
Axel Pries
Development of attributes of a robust and innovative research/ Merkmale einer robusten und innovativenForschung (MERIT)
QUEST-criteria:open questions:
• priority setting• strategies of scientific rigor• transparency and dissemination or results• participation
• intramural funding schemes, e.g. CSP, Validation fund• introduction to the doctoral/dissertation program at Charité
PRIORITY SETTINGS
Warum?
• Vorhandene Evidenz darlegen• Wissenslücken identifizieren• Bisherige Studienqualität kritisch bewerten• Identifikation von Faktoren, die die Effektivität einer Maßnahme
beeinflussen• Ableiten wichtiger Informationen hinsichtlich Design neuer
Studien• “evidence-based trial design”
- Reduce waste in future research- Reduce risk for humans and animals- Reduce risk of unnecessary enrollment of humans
Wie? (a very short introduction)
• Clinical Interventions - PICOS• Population• Intervention• Control/Comparators• Outcome• Study design
• preclinical• (a) treatment/intervention• (b) disease or condition of interest• (c) animal species/cell population studied• (d) outcome measures
MeshtermsVolltextsuchegeneral expressions
Boolsche Operanden
Filter
Wo?
• PubMed, Embase• Clinicaltrials.gov, Metaregister der WHO etc..• Eigene Vorstudien (dabei Daten nachvollziehbar darstellen)
• BMJ 2011;343:d5928 doi: 10.1136/bmj.d5928• Hooijmans et al. BMC Medical Research Methodology 2014, 14:43
http://www.biomedcentral.com/1471-2288/14/43• http://syrf.org.uk/library/• https://www.york.ac.uk/media/crd/Systematic_Reviews.pdf• https://www.cochrane.de/de/ressourcen• https://www.radboudumc.nl/en/research/radboud-technology-
centers/animal-research-facility/systematic-review-center-for-laboratory-animal-experimentation
• https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3265183/pdf/LA-11-087.pdf• https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3104815/pdf/LA-09-117.pdf
Ressourcen (Auswahl)
http://www.biomedcentral.com/1471-2288/14/43http://syrf.org.uk/library/https://www.york.ac.uk/media/crd/Systematic_Reviews.pdfhttps://www.cochrane.de/de/ressourcenhttps://www.radboudumc.nl/en/research/radboud-technology-centers/animal-research-facility/systematic-review-center-for-laboratory-animal-experimentationhttps://www.ncbi.nlm.nih.gov/pmc/articles/PMC3265183/pdf/LA-11-087.pdfhttps://www.ncbi.nlm.nih.gov/pmc/articles/PMC3104815/pdf/LA-09-117.pdf
STRATEGIES OF SCIENTIFIC RIGOR (I)
Highly effective treatmentsin animal stroke models
PLoS Biol. 2010;8(3):e1000344.
http://www.ncbi.nlm.nih.gov/pubmed/20361022
Phase III studiesshow no effect
Minnerup et al.Exp.Transl.Stroke Med.2014;6:2
PossibleSolutions
Reduce Bias!Use blinding, randomization,in/exclusion criteria.Report results according to guidelines (e.g. ARRIVE, CONSORT, PRISMA, etc.)Pre-register
Use statistics sensibly!Go beyond the bar plot! Show individual data points and distributions.
Think biological significance, think effect size.Consult a statistician.
Bias influences effect size
Alzheimer's disease models models
Blinded conduct of experiment
Blinded assessment of outcome
Blinded assessment of outcome
Stroke models (NXY-095)
Impr
ovem
ent i
n be
havi
oura
l out
com
e (S
tand
ardi
sed
Effe
ct S
ize)
0.0
0.2
0.4
0.6
0.8
1.0
1.2
Yes
No
Red
uctio
n in
infa
rct s
ize
Red
uctio
n in
infa
rct s
ize
> 30 studies > 500 animals
Toolsand Resources
Statistical Consulting (CRU and Biostats)QUEST toolbox (includes tools for Figure creation)Experimental Design Assistant for animal experimentsCourses at the QUEST Center and Promotionskolleg
STRATEGIES OF SCIENTIFIC RIGOR (II)
U N I V E R S I T Ä T S M E D I Z I N B E R L I N
Clinical Scientist
Prof. Dr. Geraldine RauchDr. Jochen Kruppa
Institute of Biometry and Clinical Epidemiology
Auswahl von Endpunkten
Endpunkte: Diejenigen Merkmale in klinischen Studien, anhand derer der Erfolg der Studie gemessen werden soll. Man unterscheidet dabei zwischen primären und sekundären Endpunkten.
Primäre Endpunkt: Erfasst dabei das Hauptziel der Studie. Dieses wird am Ende der Studie mit einem statistischen Test überprüft.
Sekundären Endpunkte: Erfassen dabei weitere Aspekte der Studie. Die Auswertung erfolgt rein deskriptiv.
Bei der Auswahl von Endpunkten sollte Folgendes beachtet werden: Angemessen für die medizinische Fragestellung Möglichst objektiv erfassbar Möglichst hohes Skalenniveau (vgl. spätere Folien)
Begriffe Zielgröße / Endpunkt: das Merkmal, das als Ergebnis einer Untersuchung interessiert, z.B. eine unter dem Einfluss der Therapie sich verändernder Laborwert oder ein Krankheitssymptom
Einflussgrößen: alle Merkmale die im funktionellen Zusammenhang zur Zielgröße stehen, z.B. bestimmte Behandlungsformen, Therapiemaßnahmen
Störgrößen/Confounder: Einflussgrößen, deren Untersuchung nicht Ziel der Studie ist, z.B. die unerwünschte Abhängigkeit vom Alter oder Geschlecht. Störgrößen sollten entweder eliminiert oder in der Analyse der Zielgrößen berücksichtigt werden. Störgrößen sind jedoch nicht immer alle erfassbar.
Harte und weiche Endpunkte
• Harte Endpunkte– lassen sich direkt erheben– Beispiele:
• Überlebenszeit• Tumoransprechen
• Weiche Endpunkte– lassen sich nur indirekt erheben– Beispiele:
• Lebensqualität• Schmerzempfinden
Studiendesign
nicht kontrolliert/einarmig
kontrolliert/mehrarmig
nicht randomisiert randomisiert
offen verblindet
monozentrisch multizentrisch
einfach
doppelt
Nicht für jede Fragestellung ist jedes Design möglich!
Goldstandard für Studien zum Wirksamkeitsnachweis
StudientypenEs gibt viele verschiedene StudientypenKlassifizierung nach unterschiedlichen Kriterien möglich:
• Fragestellung (z.B. Therapiestudie, Diagnosestudie, Prognosestudie),
• Blickrichtung (prospektiv, retrospektiv), • “Aktivität” des Forschers
– Beobachtungsstudien (Querschnittsstudien, Kohortenstudien, Fall-Kontroll-Studien)
– Interventionsstudien (nichtrandomisierte Studien, randomisierte Studie)
• Bei Arzneimittelstudien: „Phase“ der Erprobung (Phase I, II, III, IV)• usw.
Hier kein umfassender Überblick möglich!
Quelle: Dtsch Arztebl 2009; 106(15)
Literaturempfehlung
• Schumacher M, Schulgen G (2008, 3. Auflage): Methodik klinischer Studien, Springer.
• Sachs L (1993, 7. Auflage): Statistische Methoden: Planung und Auswertung, Springer.
• Weiß C (2010, 5. Auflage): Basiswissen Medizinische Statistik, Springer.• Gonick L (1993): The Cartoon Guide to Statsitics, HarperCollins Publisher
STRATEGIES OF SCIENTIFIC RIGOR (III)
Tracey L. Weissgerber, PhDTwitter: @T_Weissgerber
Reveal, Don’t Conceal: Transforming Data Visualization to Improve Transparency
Data presentation is the foundation of our collective scientific knowledge…
Figures are especially important.They often show data for key findings.
Many different data distributions can lead to the same bar graph…
Symmetric Outlier Bimodal Unequal n
Test p valueT-test: equal var. 0.035 0.074 0.033 0.051T-test: Unequal var. 0.035 0.076 0.033 0.035Wilcoxon 0.056 0.10 0.173 0.067
30
20
10
0
Weissgerber et al., PLOS Biology 2015
Why you shouldn’t use a bar graph even if your data are normally distributed
Sedentary ExerciseTrained
Sedentary ExerciseTrained
Hea
rt ra
te (b
eats
/min
)
Bar graph(mean ± SE)
Bar graphwith points
Univariatescatterplot
Range ofObserved
Values
Zone ofIrrelevance
Zone of Invisibility
0
Bar graphs1. Don’t allow you to critically evaluate continuous data2. Arbitrarily assign importance to bar height, instead of showing
how the difference between means compares to the variabilityWeissgerber et al., JBC 2017
Graphics for:- Cross sectional studies
- Experimental studies with independent groups
Dotplot Boxplot with points
Boxplot Violin plot(with or withoutpoints)
Bar graph
Outcome variable
Continuous Continuous Continuous Continuous Counts & proportions
Sample size Small Medium Large Medium to Large
Any
Data distribution
Any Any Do not use for bimodal data
Any N/A
Free violin plot tool: https://interactive-graphics.shinyapps.io/violin/
https://interactive-graphics.shinyapps.io/violin/
Free Tools for Interactive Graphics
Dot, box or violin plot: http://statistika.mfub.bg.ac.rs/interactive-dotplot/
Interactive line graph:http://statistika.mfub.bg.ac.rs/interactive-graph/
Additional resources: Twitter @T_Weissgerber
Group 1
Group 2
Group 3
Condition 1 Condition 2 Condition 30
5
10
15
20
http://statistika.mfub.bg.ac.rs/interactive-dotplot/http://statistika.mfub.bg.ac.rs/interactive-graph/
Why we need to report more than “Data were analyzed by t-tests and ANOVA”
• Meta-research studies show that statistical errors are common. These include:– Failing to specify what test was used – Using suboptimal or inappropriate tests– Incorrect p-values
• T-tests and ANOVA are the most common analysis techniques in many basic biomedical science fields
Why we need to report more than “Data were analyzed by t-tests and ANOVA”
Systematic review: Many physiology papers are missing information needed to determine what type
of ANOVA was performed
Essential Details Papers with Missing Information
Number of factors 17%
Names of factors 54%
Post-hoc tests 27%
Between vs. within-subjects factors for repeated measures ANOVA
63%
Papers rarely contain information needed to verify the test result
Essential Details Papers with Missing Information
T-tests(n = 163)
ANOVA(n = 225)
Test statistic 96% 95%
Degrees of freedom * 7% 97%
Exact p-value 69% 78%
* Exact sample size is also acceptable for t-tests
This information is essential to identify bias & correct errors
1. Confirm that the correct test was used2. Confirm test results
– Errors in reported p-value are common; may alter conclusions in 1/8 papers1
3. Assess bias: Were observations excluded without explanation?– Among papers with animal models of cancer & stroke2
• 7-8% excluded animals without explanation• 2/3 didn’t have enough information to assess
1Nuitjen et al., Behav Res Methods 20152Holman et al., PLOS Biol 2016
Solutions• Report exactly what test you used
Test ReportingT-tests Unpaired vs. paired, equal vs. unequal variance
ANOVA Number & names of factors, between vs. within subjects factors, post-hoc tests, any interaction terms included in the ANOVA
More complex tests Detail needed to reproduce analysis
These two tests…
ANOVAwithout repeated measures
Repeated MeasuresANOVA
…see the data differently
Compares3 independent groups
(n = 30, 10/group)
Compares10 pairs
of related observations
(n = 10)
…test differenthypotheses
Null hypothesis: Mean T1 = Mean T2 = Mean T3
Null hypothesis: Mean T1 = Mean T2 = Mean T3
when population means are related
…use information differently
More unexplained variability Less unexplained variability – we can account for the effect of “subject”.
Sums of squaresConditions (between groups)
Residual (within groups)
Total
240
2604
2844
Sums of squaresConditions (between groups)
Residual (within groups)SubjectsError
Total
240
26042195
409
2844
…give different results
overall p = 0.304 overall p = 0.016
Why does this matter? An example…
?
0
20
40
60
0 1 2 3T1 T2 T30
20
40
60
T1 T2 T3
Solutions• Report exactly what test you used• Improve clarity by describing simple tests in
figure & table legends• Report test statistic, degrees of freedom,
exact p-value• Use Statcheck: http://statcheck.io• Deposit code: Make your analyses
reproducibleClear reporting allows you to identify &
correct errors prior to publication!
http://statcheck.io
STAKEHOLDER ENGAGEMENT
Patient Engagement in biomedical research: From research subjects to partners in research
Engaging patients as partners and not just subjects in research can improve research! They hold important experience-based expertise from living with diseases.
Miravittles M et al. (2013), Respiratory Medicine 107, 1977-1985
Patient Engagementin biomedical research:Key areas for patient engagement
Geissler J et al. (2017), Therapeutic Innovation & Regulatory Science 51(5), 612-619.
Research phases
Key areas
Key areas
Patient Engagementin biomedical research:Important resources
Best Practice Examples Possible gatekeeper to identify patients……although so far no
institution appointed.
Other stakeholders whose involvement might benefit research:• Political actors involved in decision-making about medical
products, e.g. G-BA, IQWiG, BfArM, PEI?• Industry actors from the pharmaceutical and biotech world?• Clinicians using the research outcomes?• Training institutions teaching the findings?• …
Patient Engagementin biomedical research:Who else should be involved?
• Who should represent (and why) particular stakeholder groups?• Do representatives need particular training?• How should engagement activities be structured to ensure a level
playing field?• What conflicts of interest exist and how can they be managed?• Is there a need for a coordinating institution for patient
engagement?• …
Patient Engagementin biomedical research:Further open questions
EInführungsveranstaltung CSP 2018 �QUEST-KriterienDie folgenden Folien sind als Anregung zu verstehen,�sie ersetzen nicht die AKTIVE Auseinandersetzung mit den QUEST-Kriterien im Hinblick auf Ihre spezifische Forschungsfrage.�Bei Fragen wenden Sie sich gerne an Dr. Miriam Kip ([email protected] oder [email protected]).�QUEST Tool box: (https://www.bihealth.org/en/quest-center/mission-approaches/englische-uebersetzung/the-quest-toolbox/)��MERIT-Project�PRIORITY SETTINGSWarum?Wie? (a very short introduction) Wo?Ressourcen (Auswahl)�STRATEGIES OF SCIENTIFIC RIGOR (I)Highly effective treatments�in animal stroke modelsPhase III studies�show no effectPossible�SolutionsBias influences effect sizeTools�and Resources�STRATeGIES OF SCIENTIFIC RIGOR (II)Foliennummer 16Auswahl von EndpunktenBegriffe Harte und weiche EndpunkteStudiendesignStudientypenFoliennummer 22LiteraturempfehlungSTRATEGIES OF SCIENTIFIC RIGOR (III)��Foliennummer 25Data presentation is the foundation of our collective scientific knowledge…Many different data distributions can lead to the same bar graph…Why you shouldn’t use a bar graph even if your data are normally distributedGraphics for:�- Cross sectional studies �- Experimental studies with independent groupsFree Tools for Interactive GraphicsWhy we need to report more than “Data were analyzed by t-tests and ANOVA”Why we need to report more than “Data were analyzed by t-tests and ANOVA”Papers rarely contain information needed to verify the test resultThis information is essential to identify bias & correct errorsSolutionsWhy does this matter? An example…Solutions�STAKEHOLDER EngagementPatient Engagement �in biomedical research: �From research subjects to partners in researchPatient Engagement�in biomedical research:�Key areas for patient engagementPatient Engagement�in biomedical research:�Important resourcesPatient Engagement�in biomedical research:�Who else should be involved?Patient Engagement�in biomedical research:�Further open questions