V Jornadas eMadrid sobre “Educación Digital”. Vincent Aleven, Carnegie Mellon University: Supporting Self-Regulated Learningwith Intelligent Tutoring Systems

1July 2015 Supporting Self-Regulated Learning with Intelligent Tutoring Systems

Supporting Self-Regulated Learning with Intelligent Tutoring SystemsVincent AlevenAssociate ProfessorHuman-Computer Interaction InstitutePittsburgh Science of Learning Center (LearnLab)Carnegie Mellon University

Based on the PhD research of

Yanjin Long


Introduction

• Intelligent tutoring systems (ITSs) can support domain-level learning effectively

• How can we make them even more effective? • How can an ITS help students improve as self-

regulated learners?• Key aspects of self-regulated learning (SRL):

Self-assessment and problem selection– Important in many learning environments!

• How can an ITS support self-assessment and problem selection effectively? Do students learn better as a result?


Overview

• Cognitive Tutors• Role of self-assessment and problem selection

in self-regulated learning (SRL)• 2 classroom studies of how to support these

SRL aspects in ITSs• Discussion

Work by John Anderson, Ken Koedinger, Albert Corbett, Steve Ritter, and others


Algebra Cognitive Tutor

Use graphs, graphics calculator

Analyze real world problem scenarios

Use table, spreadsheet

Use equations, symbolic calculator

Tracked by knowledge tracing

Model tracing to provide context-sensitive Instruction


Real-world Impact of Cognitive Tutor Courses

• Spin-off company Carnegie Learning, Inc.• Over 500,000 students per year


Effectiveness of Cognitive Tutor Algebra at Scale• Over 17,000 students in

147 schools across 7 states• Duration: 2 years• Cost: $6 million• Test: Algebra Proficiency

Exam, 32-item multiple-choice assessment

• Research participants using Cognitive Tutor Algebra improved eight percentile points, compared to the control group – About the same as doubling

math learning in a year for a high-school student


• Cognitive Model: A system that can solve problems in the various ways students can

Strategy 1: IF the goal is to solve a(bx+c) = d THEN rewrite this as abx + ac = d

Strategy 2: IF the goal is to solve a(bx+c) = d THEN rewrite this as bx + c = d/a

Misconception: IF the goal is to solve a(bx+c) = d THEN rewrite this as abx + c = d

Cognitive Tutor Technology:Use ACT-R theory to individualize instruction



3(2x - 5) = 9

6x - 15 = 9 2x - 5 = 3 6x - 5 = 9


If goal is solve a(bx+c) = dThen rewrite as abx + ac = d If goal is solve a(bx+c) = d

Then rewrite as abx + c = d

If goal is solve a(bx+c) = dThen rewrite as bx+c = d/a

• Model Tracing: Follows student through their individual approach to a problem -> context-sensitive instruction

Vincent Aleven

animate one by one?

Vincent Aleven

red for bug rule?



3(2x - 5) = 9

6x - 15 = 9 2x - 5 = 3


If goal is solve a(bx+c) = dThen rewrite as abx + ac = d

• Model Tracing: Follows student through their individual approach to a problem -> context-sensitive instruction

Hint message: “Distribute a across the parentheses.” Bug message: “You need to

multiply c by a also.”

• Knowledge Tracing: Assesses student's knowledge growth -> individualized activity selection and pacing

Known? = 85% chance Known? = 45%

6x - 5 = 9

If goal is solve a(bx+c) = dThen rewrite as abx + c = d

Vincent Aleven

could leave out knowledge tracing part - even though it is kind of cool


Bayesian Knowledge Tracing drives Open Learner Models (OLMs) and Task Selection (Cognitive Mastery)


Assumptions Underlying Bayesian Knowledge Tracing

Learning assumptions• Each rule is either learned or unlearned• In problem-solving a rule can transition from

unlearned to learned at each opportunity to apply it • No forgetting - Rules do not transition back from

learned to unlearned Performance assumptions• If the rule is in the learned state there is some

chance the student will slip and make a mistake.• If the rule is in the unlearned state there is some

chance the student will guess correctly. (Corbett & Anderson, 1995; Corbett et al., 2000)


Inferring Learning State• Following each opportunity to apply a rule, the new

probability estimate that the rule has been learned, p(LN|EN), is:

p(LN|EN) = p(LN-1|EN) + (1 - p(LN-1|EN))*p(T)

Bayes Theorem

p(Ln-1|Evidencen) expands Bayes’ Theorem

p(Ln-1|Cn) = p(Ln-1)*(1-p(S))

p(Ln-1)*(1-p(S)) + p(Un-1)*p(G)

p(Ln-1|Incn) = p(Ln-1)*p(S)

p(Ln-1)*p(S) + p(Un-1)*(1-p(G))

(Corbett & Anderson, 1995; Corbett et al., 2000)


Self-Regulated Learning: Great Theoretical Diversity


Background: Self-Regulated Learning

• How do instructional intervention aimed at supporting these elements affect robust learning?

Planning• Goal Setting• Study Choice

Monitoring and Control• Self-Assessment• Help Seeking

Evaluating• Self-Explanation


Why is Self-Assessment Important?

• The process of self-assessing can facilitate deep thinking and reflection (Boud, 2004; White & Frederiksen, 1998)

• The results of self-assessment can lead to better learning plans and study choices, as well as better learning outcomes (Thiede, Anderson & Therriault, 2003; Winne & Hadwin, 1998)

• However, students’ self-assessment is often inaccurate (Dunlosky & Lipko, 2007; Nelson, 1996)


Research Questions – Study 11. (How) can the Open Learner Model be

leveraged to support student self-assessment in ITSs?

2. Does the self-assessment support lead to more accurate self-assessment and better learning outcomes?

3. Does study choice in an ITS lead to better learning outcomes, especially when combined with self-assessment support?


Geometry Cognitive Tutorwith Skill Meter


Skill Diary, Part 1


Skill Diary, Part 2


Skill Diary, Part 3


Study 1• Hypothesis: Periodically filling out structured

Skill Diaries helps students self-assess and learn better

• Participants:– 122 students from 2 teachers’ 6 classes in a local

high school– Complete data for 95 students

• Procedure: Students worked on tutor for 3 class periods (volume and surface areas for spheres and right prisms), took paper pre-test before and post-test after

• Experimental condition: Skill Diary • Control condition: Control Diary (no self-

assessment)


Control Diary


Post-Test: Experimental Group Better on Reproduction Problems

F(1, 93) = 3.86, p = .052, η² = .040

Caveat: when pre-test score is used as co-variate, the difference between two groups on reproduction problems was on the borderline of significance (F(1, 92) = 2.75, p = .101, η² = .029)


Post-test: Lower Performing Students Who Used Skill Diaries Did Better

(F(1, 44) = 4.586, p = .038, η² = .094; pre-test reproduction problem score was used as co-variate)


Measuring Self-assessment Accuracy on Pre- and Post-Tests

• Measures the discrepancy between self-assessed and actual performance.

(Schraw, 2009)


Self-Assessment AccuracyAbsolute Accuracy Index

Pre-Test Post-Test0

0.05

0.1

0.15

0.2

0.25

0.3

0.35

0.4

Lower-PerformingHigher-Performing

• Higher performing students have more accurate self-assessment

Note: Lower means more accurate


Self-Assessment Accuracy of Lower-Performing StudentsAbsolute Accuracy Index

t(23) = 2.257, p = .034

SA o

n Pr

e-Te

st

SA o

n Po

st-T

est

0

0.05

0.1

0.15

0.2

0.25

0.3

0.35

0.4

Experimental GroupControl Group

• Accuracy of SA improves from Pre to Post for lower-performing students


Used DataShop to Analyze Learning Processes (i.e., ITS Analytics)

28https://pslcdatashop.web.cmu.edu/


Process Measures

Correlations

Pre-Test

Post-Test

Number of Hints -.56** -.47**

Time Spent on Each Hint .20 .34**

Number of Incorrect Attempts

-.35** -.32**

Assistance Score -.52** -.47**

Time Spent on Each Step -.19 -.20

* p <.05

** p <.01


Process Measures

Correlations Condition Differences

Pre-Test

Post-Test

Exp Ctrl η²

Number of Hints -.56** -.47** .054 .082 .049*

Time Spent on Each Hint .20 .34** 17.5 12.4 .037*

Number of Incorrect Attempts

-.35** -.32** .085 .092 .031

Assistance Score -.52** -.47** .140 .174 .055*

Time Spent on Each Step -.19 -.20 15.4 14.4 .027

* p <.05

** p <.01


Summary of Contributions

• Skill Diaries practical way of supporting effective self-assessment for lower-performing students

• Demonstrates a beneficial role of self-assessment in students’ learning of problem-solving tasks with an ITS


Research Questions1. (How) can the Open Learner Model be










2 x 2 Experiment

• Two factors:– OLM vs. no OLM– Student control over problem selection vs.

system control (PS vs. noPS)• Pre/post test before/after with

procedural and conceptual items• Participants:

- 62 students from one teacher’s three 7th-grade classes at a middle school in Pittsburgh

- 56 students completed all five levels


Used Tutor Built with our ITS Authoring Tools

http://ctat.pact.cs.cmu.edu/


The linear equation tutor

(Waalkens, Aleven & Taatgen, 2013)


Use of Open Learner Model (OLM) to Support Self-Assessment

Students are prompted to self-assess …

At the end of each problem:

and then see the system’s update of the skill bars


Use of Open Learner Model (OLM) to Support Study Choice


Students Improved Significantly from Pre-Test to Post-Test


OLM Conditions Did Significantly Better on Post-tests

Main effect of OLM on the overall test score: p = .031, η² = .078

Main effect of OLM on the conceptual items: p = .026, η² = .082


Process Measures

Main effect of OLM on incorrect attempts (p = .062, η² = .059), on assistance score (p = .009, η² = .116);

Main effect of PS on the assistance score (p = .075, η² = .056)


Do Self-Assessment Scores Relate to Actual Test Scores?

Different way of asking if self-assessment was accurate







Students have moderate to high accuracy of self-assessment

Absolute self-assessment (SA) accuracy for each condition

[Schraw, 2009]

A lower the index means more accurate self-assessment.


Summary of the Main Results

• All students improved significantly on solving linear equations after using the tutor

• The students who had access to the OLM did significantly better on post-test

• The students had moderate to high accuracy of self-assessment


Contributions• First controlled classroom experiment that

shows an OLM significantly enhances students’ learning outcomes with an ITS

• Helps establish the important role of self-assessment in students’ learning of problem solving tasks


Final Thoughts

• Studies show that ITS technology can be extended to support self-regulated learning (SRL) skills

• Future work– Study wider range of SRL processes and

learning environments– Study influence on future learning: Did

students become better learners?– Study SRL in the context of MOOCs




Year 9: Self-assessment and study choice

1. How can we design the OLM so it effectively supports self-assessment in ITSs?




Year 10: full cycle of SRL

• Include other SRL processes, such as goal setting

• Further redesign the Linear Equation Tutor to support the SRL processes and use it as the experiment platform for classroom evaluations


Year 10: Shared OLM

• Open students’ progress information to their peers through shared OLMs

• Bull, Mabbott and Abu-Issa’s (2007) survey study pointed out the potential of shared OLMs for fostering motivation and supporting goal setting

• Few empirical studies have been conducted to investigate the effects of shared OLMs on supporting different SRL processes in ITSs, as well as the further effects on students’ learning outcomes and motivation


Year 10: Shared OLM

• Use HCI methods to investigate how to best design the shared OLM in the linear equation tutor to allow students exchange and discuss their goals, progress, specific errors made in the tutor, understanding of their own learner models, etc. with their peers

• Vary the SRL processes supported by the tutor (mainly through the shared OLM) and conduct a controlled experiment to investigate the effects with and without the shared OLM


An ITS Success CaseCognitive Tutor Algebra• Most widely used ITS

– 2,700 schools across the country– Marketed by spin-off company Carnegie

Learning– Bought by the Apollo Group (runs University of

Phoenix)• “Exemplary Curriculum” by US Dept of Ed• Field studies reported in highly cited paper

– Koedinger, K. R., Anderson, J. R., Hadley, W. H., & Mark, M. A. (1997). Intelligent tutoring goes to school in the big city. International Journal of Artificial Intelligence in Education, 8(1), 30-43.

. . .




• Tells us that cognitive tutor curriculum is more effective than other curricula, but tells us very little (if anything) about why

• Useful for decision makers, not for scientists …


Examples of Successful Metacognitive Interventions• Reading comprehension through “reciprocal

teaching” (Palincsar & Brown, 1984)• Self-assessment of science inquiry cycle

(White & Frederiksen, 1998)• Self-addressed metacognitive

questions(Mevarech & Fridkin, 2006)• Reflecting on quiz feedback (Zimmerman &

Moylan, 2009)


• Geometry example?


Our prior work: survey and interview on the use of the Skillometer

• 44 experienced Cognitive Tutor users• Students inspect the Skillometer frequently

during learning• Students don’t actively self-assess or reflect

using the Skillometer• So, not likely that simply presenting the

Skillometer supports students’ self-assessment – E.g., improving its frequency or accuracy

[Long & Aleven, 2011]


Was the Skill Diary more effective for lower-performing students?

• Previous literature suggested that good students tend to have better self-assessment accuracy [Chi et al, 1989]

• found their intervention was more helpful for lower ability students (Hartley & Mitrovic, 2002)


Prior work on self-assessment and OLM

• Hartley and Mitrovic (2002) - Compared students’ learning gains when with or

without access to an inspectable OLM- Found no significant differences between conditions

• Long & Aleven (2011) - Students inspect the OLM frequently - Students don’t actively self-assess or reflect using the OLM


View-1 of the new OLM on the problem solving screen








View-2 of the new OLM


noOLM+PS condition


OLM+noPS


noOLM+noPS


Prior Work on Supporting Self-Assessment in ITSs • Tutor that guides students through self-

assessment activities improved self-assessment on better mastered skills (Roll et al., 2011)

• Tutor that provides metacognitive prompts and feedback improved students’ self-assessment accuracy and learning efficiency (Feyzi-Behnagh, Khezri, & Azevedo, 2011)

• Does self-assessment support lead to better learning at the domain level?


Implications of the Log Data Analysis

How does the self-assessment support influence students’ learning behaviors?

1. More deliberate use of tutor help2. More careful execution of the learning task (fewer

incorrect attempts) 3. More efficient learning


Replicated Field Studies• Controlled, full year classroom experiments• Replicated over 3 years in urban schools• In Pittsburgh

& Milwaukee

• Results:50-100% better on problem solving & representation use.

15-25% better on standardized tests.

Koedinger, K. R., Anderson, J. R., Hadley, W. H., & Mark, M. A. (1997). Intelligent tutoring goes to school in the big city. International Journal of Artificial Intelligence in Education, 8(1), 30-43.


What is Metacognition?• Reasoning about one’s own thinking, learning,

memory, etc. (Brown, 1983)

• “Metacognitive learning strategies” are specific kinds/uses of metacognition that aid learning, including planning, checking, monitoring, selecting, evaluating, and revising (Schoenfeld, 1987)

• Key components of theories of self-regulated learning (SRL) (Pintrich, 2004; Winne & Hadwin, 1998; Zimmerman, 2008)

Vincent Aleven

make connection with self-regulated learning?


Objectives in Supporting Metacognition

Improve future domain learning

Improve current domain learning in the supported

environment

Improve metacognitive strategies in the supported environment

Improve future metacognitive strategies

After the metacognitive intervention

During the metacognitive interventionIdo Roll

Vincent Aleven

Take the top of the pyramid?

Vincent Aleven

illustrate which layers will be addressed in each study?


Background

• Broaden scope of SRL aspects studied • Support self-assessment in the software





Background

• Broaden scope of SRL aspects studied • Support self-assessment in the software





Why is (Student-Controlled) Problem Selection Important?

• Computer-selected problem sequences led to better learning than student-selected (Atkinson,1972)

• Less able students learned with an ITS condition that gradually increased student control (Mitrovic & Martin, 2003)

• Adaptive navigation support in QuizGuide led to increased students’ participation and better final academic performance (Brusilovsky et al., 2004)







Limitation and Future Work• Small sample size of the experiment

• The effect of study choice on learning outcomes needs further investigation

• Support other SRL processes in ITSs


What is an “Intelligent Tutoring System” (ITS)? • A kind of educational software

– Supports “learning by doing” with personalized, step-by-step guidance

• Uses cognitive modeling and artificial intelligence techniques to– Provide human tutor-like behavior– Be flexible, diagnostic & adaptive– Provide personalized instruction (e.g., select

problems on an individual basis)