Assessment of Student Learning:

Expanding the Toolbox

Sarah BunnellOhio Wesleyan University

A Framework for Thinking about Assessment

• Backward Design (Wiggins & McTighe):– 1. Identify desired results.

• What do you want to know?

– 2. Determine acceptable evidence and appropriate methods for acquiring evidence.

– 3. Then and only then… develop learning experiences and instruction.

NOTE: It’s called backward design because you start with the goal FIRST.

Different Types of “What do you want to know?” Questions

1. What works?

Do my students learn better when I do X? How would I know?

2. What is?

What is happening when students are learning or trying to learn? Where are they getting stuck?

3. What could be?

“Visions of the Possible”, What is the ideal outcome and how might I help get students there?

(From the Carnegie Academy for the Scholarship of Teaching and Learning; CASTL)

Overarching Goals of EREN1. Students will be able to apply scientific methodology (including hypothesis generation and experimental design) and recognize the importance of uncertainty for experiments at single and multiple sites.

2. Students will be able to identify factors that vary among sites across geographic or temporal scales, describe how these factors interact, and how they may affect ecological processes.  

3. Students will be able to describe the value and techniques of scientific collaboration.

4. Students will demonstrate best practices in the accurate collection, recording, and ethical management of multi-site, multi-participant datasets.

5. Students will be able to analyze, interpret, and draw conclusions from data collected in multi-site studies.

What works? What is?

What could be?

1. Students will be able to apply scientific methodology (including hypothesis generation and experimental design) and recognize the importance of uncertainty for experiments at single and multiple sites.

Revisit the Framework

• Backward Design (Wiggins & McTighe):– 1. Identify desired results.

– 2. Determine acceptable evidence and appropriate methods for acquiring evidence.

– 3. Then and only then… develop learning experiences and instruction.

Determining Acceptable Evidence – Where to look?

• Where to find evidence of Product:– Exam, Quiz, in-class performances– Pre vs. post-test shifts in learning– Group presentations, lab reports

• What about evidence of Process?

Adapted from Bass & Elmendorf, 2007

• How can we better understand these intermediate, invisible processes?

• How might we capture them? Foster them?

Novice ExpertMIRACLE

Product Product

Dimensions of Assessment

Fine Grain Holistic

What Works

What IS

What’s PossibleTemporal Dimension

Can be applied to questions of Product as well as questions of Process

What Works? Example• Impact of teaching with “Two Step Process” on

students’ ability to analyze graphical data– Pre- and post-tests; 4 institutions (240 Ss) – Competency rubric (“Improved”; “No change,

satisfactory”; “No change, unsatisfactory”; or “Worsened”)

– Improved ability to describe and create graphs – Consistent difficulty understanding IVs and

DVs., detecting trends in noisy data, and interpreting Interactions

Picone et al. (2007). Teaching Issues and Experiments in Ecology, Vol. 5

What is? Example

• Jim Sandefur - Foundations Mathematics course• Students were struggling with problem-solving• Used Video-taped think-alouds

• Observed that students who were struggling:– Got stuck right out of the gate– Didn’t try multiple strategies– They didn’t test examples

What could be? ExampleFostering Uncertainty and Community

• References to not knowing,• Rates and qualities of questions, and • Student-to student discussions

Scales of AssessmentFine-Grained Analysis Example

• Lexical Analysis (applied to discussions or journals)– Discussion Boards vs. Blogs

– Use discussion boards for clarification, and blogs for metacognitive reflection and connection

Lexical Inquiry and Word Count (LIWC; Pennebaker et al.)

***

**

p=0.10

Scales of AssessmentMid-Level Analyses

• Secondary Analysis of High-or Low-Stakes Assessments– On what content area(s) or types of questions

did most students perform well? Struggle?

– For areas of struggle: Can you identify the bottleneck?• Functional, Affective, or Conceptual

Scales of AssessmentHolistic Analyses

• Categorical judgments of performance– Capturing students’ ways of thinking (e.g.,

Marcia Baxter-Magolda)• Absolute, Transitional, Independent, and

Contextual

– Multiple Components of Performance Rubrics• Often not the same as grading rubrics• Some very good rubrics already exist for use

–AAC&U Value Rubrics

Creating your own Rubrics• Some helpful initial steps:

– As experts, complete the task to set high end– Do an initial sort to clarify categories

• Then, look for similarities within piles

• Recommend focusing on 3-5 dimensions– Goal is clear, non-double barreled

descriptions of levels of performance, from novice to expertise

– Ideal reliability between raters is ~80%

Some Final Assessment Factors to Consider

• Temporal dimension of assessment– Rapid experience sampling, semester-long

assessments, longer assessments?

• Structure of Comparison Groups– Within- vs. Between-Groups– Links back to “What do you want to know?” Qs

• Not a “clean sample”– Individual difference variables of interest?

Final Thoughts and Recommendations

• Sustainability and Scale are key– Not only to faculty quality of life– “When I changed X, Y changed.”

• Use what you already have – Meaningful, contextualized assessments– You already have a TON of data on student

learning… don’t throw it away!

Questions? Comments?Reflections?