Upload
dangnhu
View
217
Download
4
Embed Size (px)
Citation preview
© NAC / MCC 2013
André F. De Champlain, PhD, Acting Director of Research & Development
Paper presented at the 2013 IPAC Conference
Can a Standardized Set of Assessments and Markers Predict Family Medicine Specialty Certification Status?
Wednesday, October 2nd , 2013Queenstown, New Zealand
Acknowledgements
2
• Carlos Brailovsky (CFPC)
• Sirius Qin
• Marguerite Roy, PhD
• Steve Slade (CAPER)
• Sydney Smee, PhD
• Cindy Streefkerk
• Fang Tian
• Claire Touchie, MD
• 25% of practising physicians in Canada are international medical graduates (IMGs)
• Despite more training positions, we still rely on IMGs
• Rural & remote areas are the most underserviced
Need for IMGs in Canada
3
External Drivers
4
• Agreement on Internal Trade (AIT – 1994)
• AIT aims to reduce barriers to the movement of persons, goods, services and investments within Canada
• Need to develop comparable licensing across Canada to support IMG physician mobility
• Report of the Canadian Task Force on Licensure of International Medical Graduates (2004)
• Standardize licensure processes and requirements
National Assessment Collaboration (NAC)
• The NAC is an alliance of Canadian organizations streamlining the evaluation process for IMGs seeking a licence to practise medicine in Canada
• 2 main focuses
• NAC Exam (since 2011)
• OSCE which assesses readiness of IMGs for entry into Canadian residency programs
• Provides residency program directors with objective information on the skills, attitudes and knowledge level of applicants
© NAC / MCC 2013
5
National Assessment Collaboration (NAC)• The NAC is an alliance of 18 members representing
organizational & jurisdictional stakeholders across Canada
• Mandated to streamline the evaluation process for IMGs seeking a licence to practise medicine in Canada
• 2 main focuses
• Practice Ready Assessment (PRA)
• Partners from across the country are developing a pan-Canadian practice ready assessment process for international medical graduates (IMGs)
• Route available to IMGs seeking a provisional licence to enter independent practice
• Currently, 7 PRA programs across the country
© NAC / MCC 2013
6
Initial Screening(MCCEE/CLBA/
Credentials)
Practice Ready Assessment
Provisional Licensure
IMG Orientation
Summative Assessment
Licensure
(Full)
Practice Eligible Route
for Certification
Seek Alternative Career Path
Seek Alternative Career Path
Canadian
Residency
Training Required
Selection Decision
Other Routes
(i.e., Credentials (Accredited
Qualifications), Other Programs)
Clinical Assessment
(NAC OSCE)
Supervised (Monitored)
Practice
Competencies
Entry-to-Residency
Entry-to-Practice
7
Canadian
• LMCC
• Certification
IMG Physician Routes to Practice!
Assessments
(Over-time)
Common
Co
mp
arab
le P
roce
sse
sFuture: Pan-Canadian PRA Process
Practice Ready Route
Selection AssessmentsPractice-
ReadyDecision
•Minimum Eligibility•Sponsorship (potential future job)•Ranking Tools
IMG Orientation
Over-Time
•Common Toolkit•Assessor Training
Competencies & Context
Screening
Screening assessments (Point-in-Time)
ExperienceCredentials
IMG PGT not in Canada and/or has practised outside of Canada
8
NAC-PRA Predictive Validity Study
9
• Goal• Assess the extent to which a common
set of screening tools and markers are useful in predicting a number of outcomes for a cohort of Family Medicine PRA candidates including:• P/F status on the SOO and SAMPS
components of the College of Family Physicians of Canada (CFPC) certification exam
• Scores on the SOO and SAMPS components of the College of Family Physicians of Canada (CFPC) certification exam
• Time to passing of SOO&SAMPS
NAC-PRA Predictive Validity Study
10
• Usefulness of Study• Will provide valuable information with
respect to informing a pan-Canadian PRA strategy • How might we develop a framework which
concurrently allows for (1) standardization across PRA programs (via common screening tools) and; (b) tailoring of over-time (practice based) assessment tools to address local needs and requirements?
• Allow PRA programs to allocate resources to these tailored assessments
Sample - Eligibility
11
• Criteria
• IMGs who were admitted to a NAC PRA program between 2007-2011
• IMGs who had attempted the CFPC certification examination by end of 2012
• IMGs who had complete records on all variables of interest
Measurement Instruments
12
• MCC Evaluating Exam (MCCEE)• Four-hour CBT examination
composed of 180 single-best answer MCQs
• Offered at >500 sites in 80 countries
• Eligibility pre-requisite for the MCC Qualifying Exams (IMGs & US DOs)
• 50-500 score range (M~271; SD~50)
• Cut-score = 250
Measurement Instruments
13
• MCC Qualifying Exam Part I (MCCQE Part I)
• MCCQE Part I administered in 2 multi-week windows at >12 dedicated secure sites across Canada
• First part is composed of 196 single-best answer computer-delivered MCQs (3.5 hours – 75% of composite score)
• Second part contains ~ 60 clinical decision-making cases (CDMs) – SMs and write-ins (4.0 hours – 25% of composite score)
• Cut-score = 390; score range: 50-950
• Requirement for enrollment into the Canadian Medical Registry as a licentiate of the Medical Council of Canada (LMCC)
Measurement Instruments
14
• CFPC Certification Exam in Family Medicine (pre-harmonization) included:• Five 15-minute simulated office oral exams (SOOs)
• Assesses both the definition and management of health problems in a patient-centered approach
• Physician examiner acts both as patient and examiner
• 30-40 short-answer management problems (SAMPs)
• Measure a candidate’s problem solving skills and knowledge in the context of a clinical situation (3-4 questions/case)
• Overall %-correct score was computed
Analyses
15
• Predictors• MCCEE score (most recent attempt)• MCCQE Part I score (most recent attempt)• Gender • Age at start of PRA process• Years since obtaining MD degree at start of PRA• Degree at English language medical school?
• Outcomes• CFPC P/F status on SOO and SAMPS components• CFPC score on SOO and SAMPS components• Time to successful completion of CFPC SOO & SAMPS
• Models• Logistic regression models• Multiple linear regression models• Survival data analysis model (logit-linear model)
Results - Overview
16
• Sample characteristic “highlights” • N=132
• Gender: Females (51 or 38.6%); Males (81 or 61.4%)
• MD in English: Yes (45 or 34.1%); No (87 or 65.9%)
• Age at entry into PRA: Mean=40.86; SD=6.66
• Years from MD to entering PRA: Mean=16.15; SD=6.49
• Sample performance “highlights” • EE score: Mean=317.3; SD=42.9
• MCCQE Part I: Mean=469.7; SD=73.9
• SOO: Mean=59.32; SD=6.3
• SAMPS: Mean=67.04; SD=5.7
Results – Predicting SOO
17
• Predicting SOO P/F status
• Very weak predictive model
• None of the IVs were significantly associated with passing or failing the SOO component of the CFPC exam
• Predicting SOO %-correct scores
• Very weak predictive model (R2=0.07)
• Only gender significantly predicted SOO scores
• On average women outperformed men by about 3.3% (1/2 SD)
Results – Predicting SAMPS P/F –Final Model
18
Effect Chi-square Type I error Odds ratio
Gender 5.60 0.02 0.194
MCCQE Part 1 score 6.21 0.01 1.020
Age at entry into PRA 4.44 0.03 0.917
Pass rate: 78%
Model fit: L2=24.03, p<0.0001
Results – Predicting SAMPS P/F by QE1 Section – Final Model
19
Effect Chi-square
Type I error Odds ratio
Gender 6.32 0.01 0.177
MCCQE Part 1 MCQ score 5.92 0.01 1.016
Age at entry into PRA 3.98 0.05 0.922
Pass rate: 78%
Model fit: L2=22.74, p<0.0001
Results – Predicting SAMPS Scores Final Model
20
Effect Parameter estimate
t-value Type I error
Gender -2.57 -3.19 0.002
MCCQE Part 1 score 0.04 7.49 0.0001
Age at entry into PRA -0.17 -2.84 0.005
Adjusted R-squared: 0.38
Model fit: F=27.91, p=<0.0001
Results – Predicting SAMPS Scores by QE1 Section – Final Model
21
Effect Parameter estimate
t-value Type I error
Gender -2.51 -3.15 0.002
MCCQE Part 1 MCQ score 0.03 5.73 0.0001
MCCQE Part 1 CDM score 0.02 3.28 0.001
Age at entry into PRA -0.18 -3.03 0.003
Adjusted R-squared: 0.40
Model fit: F=22.60, p=<0.0001
Results – Predicting Time to Attaining SOO & SAMPS P/F Status
22
• Time to SOO P/F status
• Gender & number of attempts are significant predictors time to SOO P/F status
• Better performance on the 2nd attempt
• Time to SAMPS P/F status
• Gender & MCCQE Part I (MCQ) score significantly predict time to passing SAMPS
• Number of attempts is not a significant predictor of SAMPS P/F status
What Does This All Mean? Examples
23
Gender Age MCCQE Part 1 Score
Probability of passing SAMPS
Predicted SAMPS score
Male 30 290 0.20 60.69%
Male 30 390 0.65 64.69%
Male 30 490 0.93 68.69%
Male 50 290 0.04 57.29%
Male 50 390 0.23 61.29%
Male 50 490 0.69 65.29%
What Does This All Mean? Examples –Part Deux
• For each increase of 1 point on the MCCQE Part I (50-950), my odds of passing the CFPC SAMPS section increase by 2%
• If my colleague obtains a score of 350 on the MCCQE Part I and I obtain a score of 500, my odds of passing the SAMPS component increase by 300%!
• For each increase of 1 point on the MCCQE Part I (50-950), my SAMPS %-correct score increases by 0.04%
• If my colleague obtains a score of 350 on the MCCQE Part I and I obtain a score of 500, my SAMPS score will be 6% higher than his
© NAC / MCC 2013
24
Conclusions
25
• Weak predictive models for CFPC SOO component
• Gender predicts SOO score (again, very weak model)
• Gender and number of attempts significant predictors of time to passing the SOO component
• Candidates do better on their 2nd SOO attempt
Conclusions
26
• SAMPS component of the CFPC exam• Very strong predictive model
• MCCQE Part I scores, gender and age at entry into PRA are strong predictors of SAMPS P/F status & score• MCQ component of MCCQE Part I strong predictor of
SAMPS P/F status
• Both MCQ and CDM components predict SAMPS scores
• Gender and MCCQE Part I scores are strong predictors of time to passing SAMPS
• # of attempts does not predict SAMPS P/F status
Implications
27
• MCC EE scores do not significantly predict P/F status nor performance on the CFPC exam
• Caveat: Restriction of range effect on EE scores!
• Consider using MCCQE Part I scores as one common screening tool to predict “knowledge/problem solving skills” competency of NAC PRA candidates
• Use other predictors judiciously!
• Female and younger PRA candidates do better than older male applicants on knowledge/problem solving competencies
Implications
28
• Need an additional common, standardized screening tool to capture SOO competencies
• NAC examination?
Next Steps
29
• Further explore relationships of screening tools to PRA outcomes• Studies done by program due to different outcomes -
aggregate program-level results to better inform a pan-Canadian process
• Further establish screening models• NAC OSCE Discrimination study
• Focus on replicating research with the CFPC’s New Certification Examination in Family Medicine & other specialties
Thank You
30