How Do You Know? How Do You Know?

Assessing LearningAssessing Learning

Richard H. HershRichard H. Hersh

EFFICACYEFFICACY

HIGH, CLEAR, and SHARED EXPECTATIONSHIGH, CLEAR, and SHARED EXPECTATIONS

HIGH ACADEMIC ENGAGED TIMEHIGH ACADEMIC ENGAGED TIMETIGHTLY COUPLED CURRICULUMTIGHTLY COUPLED CURRICULUMFREQUENT, APPROPRIATE FREQUENT, APPROPRIATE ASSESSMENT and FEEDBACKASSESSMENT and FEEDBACKPOWERFUL TEACHINGPOWERFUL TEACHING

CUMULATIVE EFFECTCUMULATIVE EFFECT

Blooms TaxonomyBlooms Taxonomy

evaluationevaluation

synthesissynthesis

analysisanalysis

applicationapplication

comprehensioncomprehension

knowledgeknowledge

How Do You Know?How Do You Know?

CONSTRUCTCONSTRUCT– Critical thinkingCritical thinking

The ability to……..The ability to……..

PERFORMANCEPERFORMANCE

Write a memo to Write a memo to your boss that……..your boss that……..

ASSESSMENT AS FORM OF ASSESSMENT AS FORM OF TEACHING AND LEARNINGTEACHING AND LEARNING

Assessment results linked to objectives:Assessment results linked to objectives:

Timely feedbackTimely feedback

Appropriate feedbackAppropriate feedback

Formative/diagnosticFormative/diagnostic

SummativeSummative

Performance as well as knowledgePerformance as well as knowledge

AUTHENTIC ASSESSMENTAUTHENTIC ASSESSMENT

WRITTENWRITTEN

ORALORAL

PEER ASSESSMENTPEER ASSESSMENT

TIGHTLY COUPLED CURRICULUMTIGHTLY COUPLED CURRICULUM

OBJECTIVES CURRIOBJECTIVES CURRIculum culum PEDAGOGY ASSESSMENTPEDAGOGY ASSESSMENT

TEACHING TO A TEST WORTH TESTING TOTEACHING TO A TEST WORTH TESTING TO

objectives assessment curriculum pedagogyobjectives assessment curriculum pedagogy

Three types of assessmentThree types of assessment

NORM REFERENCENORM REFERENCE

STANDARDS-BASEDSTANDARDS-BASED

VALUE ADDEDVALUE ADDED

COLLECTIVE OUTCOMESCOLLECTIVE OUTCOMES

Critical thinkingCritical thinkingAnalytical reasoningAnalytical reasoningProblem solvingProblem solvingWritingWriting

Everybody’s business is nobody’s businessEverybody’s business is nobody’s business

Everybody’s business Everybody’s business isis everybody’s business everybody’s business

Evaluation of evidenceEvaluation of evidence

How well does the student assess the quality and How well does the student assess the quality and relevance of evidence?relevance of evidence? Does the student determine what information is or is not Does the student determine what information is or is not pertinent to the task at hand? pertinent to the task at hand? Does the student distinguish between rational claims and Does the student distinguish between rational claims and emotional ones, fact from opinion? emotional ones, fact from opinion? Can the student recognize the ways in which the Can the student recognize the ways in which the evidence might be limited or compromised? evidence might be limited or compromised? Does the student spot deception and holes in the Does the student spot deception and holes in the arguments of others? arguments of others? Has the student considered all sources of evidence?Has the student considered all sources of evidence?

Analysis and Synthesis of EvidenceAnalysis and Synthesis of Evidence

How well does the student analyze and synthesize data and How well does the student analyze and synthesize data and information?information? Does the student present their own analysis of the data or Does the student present their own analysis of the data or information, or do they simply present it as is? information, or do they simply present it as is? Does the student commit or fail to recognize logical flaws in an Does the student commit or fail to recognize logical flaws in an argument. (e.g. does the student understand the distinction argument. (e.g. does the student understand the distinction between correlation and causation?)between correlation and causation?)Does the student break down the evidence into its component Does the student break down the evidence into its component parts? parts? Does the student draw connections between discrete sources of Does the student draw connections between discrete sources of data and information? data and information? How does the student deal with conflicting, inadequate, or How does the student deal with conflicting, inadequate, or ambiguous information? ambiguous information?

Drawing ConclusionsDrawing Conclusions

How well does the student form a conclusion from their analysis?How well does the student form a conclusion from their analysis? Is the student’s argument logically sound? Is the student’s argument logically sound? Is it rooted in data and information rather than speculation and Is it rooted in data and information rather than speculation and opinion? opinion? Does the student choose the strongest set of data to support his or Does the student choose the strongest set of data to support his or her argument? her argument? Does the student prioritize in his or her argumentation? Does the student prioritize in his or her argumentation? Does the student overstate, or understate, his or her conclusions? Does the student overstate, or understate, his or her conclusions? Can the student identify holes in the evidence, and subsequently Can the student identify holes in the evidence, and subsequently suggest additional information that might resolve the issue? suggest additional information that might resolve the issue?

Acknowledging alternative explanations/viewpointsAcknowledging alternative explanations/viewpoints

How well does the student consider other options and How well does the student consider other options and acknowledge that their answer is not the only acknowledge that their answer is not the only perspective?perspective?Does the student recognize that the problem is complex Does the student recognize that the problem is complex with no clear answer?with no clear answer?Does the student bring up other options and weigh them Does the student bring up other options and weigh them in their decision.in their decision.Does the student consider all stakeholders or affected Does the student consider all stakeholders or affected parties in suggesting a course of action?parties in suggesting a course of action?Does the student qualify their response and Does the student qualify their response and acknowledge the need for additional information in acknowledge the need for additional information in making an absolute determination.making an absolute determination.

WritingWriting

Analytic writing skills are invariably dependent on clarity Analytic writing skills are invariably dependent on clarity of thought. Therefore, analytic writing and critical of thought. Therefore, analytic writing and critical thinking, analytic reasoning, and problem solving are thinking, analytic reasoning, and problem solving are related skills sets. The CLA measures critical thinking related skills sets. The CLA measures critical thinking performance by asking students to explain their rationale performance by asking students to explain their rationale for various conclusions in writing. In doing so, their for various conclusions in writing. In doing so, their performance is dependent on both writing and critical performance is dependent on both writing and critical thinking as integrated rather than separate skills. We thinking as integrated rather than separate skills. We evaluate writing performance using holistic scores that evaluate writing performance using holistic scores that consider several aspects of writing depending on the consider several aspects of writing depending on the task. task.

PresentationPresentation

How clear and concise is the argument?How clear and concise is the argument? Does the student clearly articulate the Does the student clearly articulate the argument? argument? Does the student clearly articulate the Does the student clearly articulate the context for that argument? context for that argument? Is the evidence used to defend the Is the evidence used to defend the argument correct and precise? argument correct and precise? Is the evidence presented in a Is the evidence presented in a comprehensible and coherent fashion?comprehensible and coherent fashion?

DevelopmentDevelopmentHow effective is the structure? How effective is the structure? Is the organization of the argument logical? Is it Is the organization of the argument logical? Is it cohesive? cohesive? Are there any gaps in the development of the Are there any gaps in the development of the argument? argument? Are there any significantly extraneous elements Are there any significantly extraneous elements in the argument’s development? in the argument’s development? In what order is the evidence presented, and In what order is the evidence presented, and how does that structure contribute to the how does that structure contribute to the persuasiveness and coherence of the persuasiveness and coherence of the argument?argument?

PersuasivenessPersuasiveness

How well does the student defend the argument?How well does the student defend the argument? What evidence is presented in support of the argument, What evidence is presented in support of the argument, and how effectively does the student present it? and how effectively does the student present it? Does the student draw thoroughly and extensively from Does the student draw thoroughly and extensively from the available range of evidence? the available range of evidence? How well does the student analyze that evidence? How well does the student analyze that evidence? Does the student consider counterarguments, and Does the student consider counterarguments, and address weaknesses in his/her own argument?address weaknesses in his/her own argument?

MechanicsMechanics

What is the quality of the student’s writing?What is the quality of the student’s writing? Is vocabulary and punctuation used correctly? Is vocabulary and punctuation used correctly? effectively? effectively? Is the student’s understanding of grammar Is the student’s understanding of grammar strong? strong? Is the sentence structure basic, or more complex Is the sentence structure basic, or more complex and creative? and creative? Does the student use proper transitions? Does the student use proper transitions? Are the paragraphs structured logically and Are the paragraphs structured logically and effectively?effectively?

InterestInterest

How well does the student maintain the reader’s How well does the student maintain the reader’s interest?interest? Does the student use creative and engaging Does the student use creative and engaging examples or descriptions?examples or descriptions?Does the structure syntax and organization add Does the structure syntax and organization add to the interest of their writing?to the interest of their writing?Do they use colorful but relevant metaphors, Do they use colorful but relevant metaphors, similes etc.?similes etc.?Does the writing engage the reader? Does the writing engage the reader? Does the writing leave the reader thinking?Does the writing leave the reader thinking?

CLA’s MeasuresCLA’s Measures

Performance tasks (90 minutes)Performance tasks (90 minutes)– Document basedDocument based– Contextualized questionsContextualized questions– Split screen/dialogue box formatSplit screen/dialogue box format– Analytic and holistic scoringAnalytic and holistic scoring

Analytic writing (essay) promptsAnalytic writing (essay) prompts– Make-an-argument (45 minutes)Make-an-argument (45 minutes)– Break-an-argument (30 minutes)Break-an-argument (30 minutes)

Background questionnaireBackground questionnaire

Performance TasksPerformance Tasks90-minute real life problems90-minute real life problemsGeneral directions and context General directions and context Need to combine information from different types of documentsNeed to combine information from different types of documentsA few open-ended questions, no one “right” answer—must explain A few open-ended questions, no one “right” answer—must explain rationalerationaleSplit screenSplit screen

Right: Document LibraryRight: Document Library Left: Question and answer blockLeft: Question and answer block

The TaskThe Task

Jamie Eager is a candidate who is opposing Mayor Jamie Eager is a candidate who is opposing Mayor Pat Stone for reelection. Eager critiques the Pat Stone for reelection. Eager critiques the Mayor’s solution to reducing crime by increasing Mayor’s solution to reducing crime by increasing the number of police officers. Eager proposes the the number of police officers. Eager proposes the city support a drug education program for addicts city support a drug education program for addicts because, according to Eager, addicts are the because, according to Eager, addicts are the major source of the city’s crime problem. major source of the city’s crime problem.

Mayor Pat Stone asks you to do two things (1) Mayor Pat Stone asks you to do two things (1) evaluate the validity of Eager’s proposal and (2) evaluate the validity of Eager’s proposal and (2) assess the validity of Eager’s criticism of the assess the validity of Eager’s criticism of the mayor’s plan to increase the number of officers. mayor’s plan to increase the number of officers.

The DocumentsThe Documents

Mayor Stone hands you a folder Mayor Stone hands you a folder containing various documents related containing various documents related to this matter, but warns you that some to this matter, but warns you that some of them may not be more relevant than of them may not be more relevant than others. Your task is to review these others. Your task is to review these materials and respond to the mayor’s materials and respond to the mayor’s request in preparation for tomorrow request in preparation for tomorrow night’s public debate with Eager.night’s public debate with Eager.

Newspaper ArticleNewspaper Article

MemoMemo

Crime StatisticsCrime Statistics

Research BriefResearch Brief

Research AbstractsResearch Abstracts

Crime Rates ChartCrime Rates Chart

Crime and Drug Use TablesCrime and Drug Use Tables

Crime StatisticsCrime Statistics

CLA Tasks (2)CLA Tasks (2)

Analytic writing tasks (45 and 30 minutes)Analytic writing tasks (45 and 30 minutes)

Measure ability to articulate complex ideas, examine Measure ability to articulate complex ideas, examine claims and evidence, support ideas with relevant claims and evidence, support ideas with relevant reasons and examples, sustain a coherent discussion, reasons and examples, sustain a coherent discussion, and use standard written English.and use standard written English.

– Make-an-Argument (45 minutes): support or reject a position on Make-an-Argument (45 minutes): support or reject a position on some issuesome issue

– Critique-an-Argument (30 minutes): evaluate the validity of an Critique-an-Argument (30 minutes): evaluate the validity of an argument made by someone else.argument made by someone else.

Analytic Writing TasksAnalytic Writing Tasks

Make-An-Argument PromptMake-An-Argument Prompt

“ “In our time, specialists of all kinds In our time, specialists of all kinds are highly overrated. We need more are highly overrated. We need more generalists – people who can generalists – people who can provide broad perspectives.”provide broad perspectives.”

Directions: 45 Minutes, agree or Directions: 45 Minutes, agree or disagree and explain the reasons for disagree and explain the reasons for your position. Student selects one of your position. Student selects one of two prompts to answer.two prompts to answer.

STEP 1-MULTIPLE CHOICESTEP 1-MULTIPLE CHOICE

The test is designed to measure basic The test is designed to measure basic science knowledge. Some questions test science knowledge. Some questions test the examinee's fund of information per se, the examinee's fund of information per se, but the majority of questions require the but the majority of questions require the examinee to interpret graphic and tabular examinee to interpret graphic and tabular material, to identify gross and microscopic material, to identify gross and microscopic pathologic and normal specimens, and to pathologic and normal specimens, and to solve problems through application of solve problems through application of basic science principles. basic science principles.

STEP 1 EXAM QUESTIONSTEP 1 EXAM QUESTION

Example Question 1Example Question 1 A 32-year-old woman with type 1 diabetes A 32-year-old woman with type 1 diabetes mellitus has had progressive renal failure over the past 2 years. She mellitus has had progressive renal failure over the past 2 years. She has not yet started dialysis. Examination shows no abnormalities. has not yet started dialysis. Examination shows no abnormalities. Her hemoglobin concentration is 9 g/dL, hematocrit is 28%, and Her hemoglobin concentration is 9 g/dL, hematocrit is 28%, and mean corpuscular volume is 94 µm3. A blood smear shows mean corpuscular volume is 94 µm3. A blood smear shows normochromic, normocytic cells. Which of the following is the most normochromic, normocytic cells. Which of the following is the most likely cause?likely cause?

A.Acute blood lossB.Chronic lymphocytic leukemiaC.Erythrocyte A.Acute blood lossB.Chronic lymphocytic leukemiaC.Erythrocyte enzyme deficiencyD.Erythropoietin enzyme deficiencyD.Erythropoietin deficiencyE.ImmunohemolysisF.Microangiopathic deficiencyE.ImmunohemolysisF.Microangiopathic hemolysisG.Polycythemia veraH.Sickle cell diseaseI.Sideroblastic hemolysisG.Polycythemia veraH.Sickle cell diseaseI.Sideroblastic anemiaJ.b-Thalassemia trait (Answer D)anemiaJ.b-Thalassemia trait (Answer D)

STEP 2-MULTIPLE CHOICESTEP 2-MULTIPLE CHOICE Test questions focus on the principles of clinical science that are deemed important for the practice Test questions focus on the principles of clinical science that are deemed important for the practice

of medicine under supervision in postgraduate training. of medicine under supervision in postgraduate training.

Normal Conditions and Disease categories (Dimension 1) form the main axis for organizing the Normal Conditions and Disease categories (Dimension 1) form the main axis for organizing the outline. The first section deals with normal growth and development, basic concepts, and general outline. The first section deals with normal growth and development, basic concepts, and general principles. The remaining sections deal with individual disorders. principles. The remaining sections deal with individual disorders.

Sections focusing on individual disorders are subdivided according to Physician Task (Dimension 2). Sections focusing on individual disorders are subdivided according to Physician Task (Dimension 2). The first set of physician tasks, Promoting Preventive Medicine and Health Maintenance, The first set of physician tasks, Promoting Preventive Medicine and Health Maintenance, encompasses the assessment of risk factors, appreciation of epidemiologic data, and the application encompasses the assessment of risk factors, appreciation of epidemiologic data, and the application of primary and secondary preventive measures. of primary and secondary preventive measures.

The second set of tasks, Understanding Mechanisms of Disease, encompasses etiology, The second set of tasks, Understanding Mechanisms of Disease, encompasses etiology, pathophysiology, and effects of treatment modalities in the broadest sense. pathophysiology, and effects of treatment modalities in the broadest sense.

The third set of tasks, Establishing a Diagnosis, pertains to interpretation of history and physical The third set of tasks, Establishing a Diagnosis, pertains to interpretation of history and physical findings and the results of laboratory, imaging, and other studies to determine the most likely findings and the results of laboratory, imaging, and other studies to determine the most likely diagnosis or the most appropriate next step in diagnosis. diagnosis or the most appropriate next step in diagnosis.

The fourth set of tasks, Applying Principles of Management, concerns the approach to care of The fourth set of tasks, Applying Principles of Management, concerns the approach to care of patients with chronic and acute conditions in ambulatory and inpatient settings. Questions in this patients with chronic and acute conditions in ambulatory and inpatient settings. Questions in this category will focus on the same topics covered in the diagnosis sections. category will focus on the same topics covered in the diagnosis sections.

STEP 2 EXAM QUESTIONSTEP 2 EXAM QUESTION

Example Items 2-3: Matching setExample Items 2-3: Matching set A.Chronic lymphocytic A.Chronic lymphocytic leukemiaB.Drug reaction C.Hodgkin disease D.Infectious leukemiaB.Drug reaction C.Hodgkin disease D.Infectious mononucleosis E.Metastatic carcinoma F.SarcoidosisG.Systemic mononucleosis E.Metastatic carcinoma F.SarcoidosisG.Systemic lupus erythematosus H.ToxoplasmosisI.TuberculosisJ.TularemiaFor lupus erythematosus H.ToxoplasmosisI.TuberculosisJ.TularemiaFor each patient with lymphadenopathy, select the most likely diagnosis. each patient with lymphadenopathy, select the most likely diagnosis. A previously healthy 30-year-old man has had fever, night sweats, A previously healthy 30-year-old man has had fever, night sweats, pruritus, and an enlarging lump above his left clavicle for 3 weeks. pruritus, and an enlarging lump above his left clavicle for 3 weeks. Examination shows a 3-cm, nontender, rubbery, supraclavicular Examination shows a 3-cm, nontender, rubbery, supraclavicular lymph node. An x-ray of the chest shows mediastinal lymph node. An x-ray of the chest shows mediastinal lymphadenopathy. lymphadenopathy. (Answer C) (Answer C) A 41-year-old woman comes to the physician for a follow-up A 41-year-old woman comes to the physician for a follow-up examination. She has taken aspirin for chronic headaches and examination. She has taken aspirin for chronic headaches and phenytoin for a seizure disorder for 2 years. Examination shows phenytoin for a seizure disorder for 2 years. Examination shows mild epigastric tenderness and bilateral, 3-cm, nontender axillary mild epigastric tenderness and bilateral, 3-cm, nontender axillary lymph nodes. A lymph node biopsy shows hyperplasia. lymph nodes. A lymph node biopsy shows hyperplasia. (Answer B) (Answer B)

Step 3 Exam-SimulationsStep 3 Exam-Simulations

Primum Computer-based Case Simulations (CCS)

Each Primum case is a dynamic, interactive simulationof a patient-care situation designed to evaluate yourapproach to clinical management, including diagnosis,treatment, and monitoring. The cases provide a meansfor observing your application of medical knowledge ina variety of patient care situations and settings overvarying periods of simulated time.