Student Efficacy: Are They Learning? Rapid Fire Panel, CNX 2014 1 April 2014

Dr. Beck Pitt The Open University

Background

• 2-year research project based at The Open University (UK)

• Funded by William & Flora Hewlett Foundation

• Aiming to build the most comprehensive picture of OER impact

• Eleven research hypotheses

• Collaboration model across different educational sectors

• Fellowship Scheme

• Global reach but with a US focus

• Practicing openness: CC-BY licensed research instruments / SOO Course / Impact Map

OER Research Hub

oerresearchhub.org #oerrhub @OER_Hub

Keyword Hypothesis

Performance OER improve student performance/satisfaction

Openness People use OER differently from other online materials

Access OER widen participation in education

Retention OER can help at-risk learners to finish their studies

Reflection OER use leads educators to reflect on their practice

Finance OER adoption brings financial benefits for students/institutions

Indicators Informal learners use a variety of indicators when selecting OER

Support Informal learners develop their own forms of study support

Transition OER support informal learners in moving to formal study

Policy OER use encourages institutions to change their policies

Assessment Informal assessments motivate learners using OER

Collaboration Model

Collaboration Model

Collaboration Model

Collaboration Model

OER Impact Map

OER Impact Maphttp://chaos.open.ac.uk

OER Impact Maphttp://chaos.open.ac.uk

Methodology

Educator and Student Surveys

Overview

OER improve student performance/satisfaction

People use OER differently from other online materials

OER use leads educators to reflect on their practice

OER adoption brings financial benefits for students/institutions

Spring-Summer 2013

Developing Questionnaires with OpenStax College

IRB Process

Autumn 2013-Christmas 2013

Surveys deployed

Ongoing

Interviews with educators and students

Comparative impact data / Stories of Impact

Adding evidence to the Impact Map

Jan-Apr 2014

Data analysis and Dissemination of findings

Interviews with Educators

Student Survey

In total 63 usable survey responses were recorded:• OpenStax Newsletter Student User Survey (50) Incentivised • College One (3)• College Two (10)

44 out of the 63 respondents have used, or currently use, OpenStax College textbooks:• OpenStax Newsletter Student User Survey (39) • College One (2) • College Two (3)

Educator Survey

In total 97 usable survey responses were recorded:• OpenStax Adopter Email list (52)• OpenStax Newsletter Educator Survey (41)* Incentivised • Individual Educators from direct introductions (4)

82 out of the 97 respondents have used, or currently use, OpenStax College textbooks:• OpenStax Adopter Email list (47)• OpenStax Newsletter Educator Survey (31)• Individual Educators from direct introductions (4)

Student Sample

• 44 Respondents in total (have used/use OSC textbooks) • 56.8% Male (n=25) and 43.2% Female (n=19)• 86.0% of respondents live in the United States (n=37)• Other respondents live in Germany, Canada, Hong Kong, UK & India (14.0%, n=6) • 65.9% of respondents have English as their first spoken language (n=29)• Just under 5% of respondents consider themselves to have a disability (4.5%, n=2) • Just under 70% of respondents were between 15-18 years old (68.3%, n=28) • Nearly 60% of respondents told us they are studying at School (59.1%, n=26). • Over 60% of respondents reported having either a High School Diploma or No Formal

Qualifications (n=10, 28.6% and 34.3%, n=12 respectively)• Just under 10% of respondents who use OSC textbooks receive any form of financial

aid (9.5%, n=4)

Educator Sample

• 82 Respondents in total (have used/use OSC textbooks) • 75.3% Male (n=61) and 24.7% Female (n=20)• Over 85% of respondents live in the United States (86.4%, n=70)• Other respondents live in Canada, Singapore, Malaysia & Australia (13.6%, n=11) • 90.0% of respondents have English as their first spoken language (n=72)• Over 90% of respondents have a Masters degree and/or PhD (38.3%, n=31, 54.3%,

n=44 respectively) • Over 50% of respondents work in a HE/University context (57.3%, n=47) • 78.0% of respondents do F/T face-to-face teaching (n=64) • Nearly 70% of respondents have been teaching for more than 10 years (67.5%, n=54)

OER Behaviours

Student Overview

• Over a quarter of respondents studied their subject via OER before joining their course (26.8%, n=11).

• 17.5% of respondents told us that their use of OER influenced their decision to register for their current course of study (n=7)

• Just under a third of respondents told us that they have adapted OER to fit their needs (31.0%, n=13)

• The biggest challenge for respondents when using OER was knowing where to find resources (61.0%, n=25)

Student Overview

• 75.0% of respondents first became aware of OpenStax College Textbooks via their instructor/course (n=30).

• 54.5% of respondents currently use the Biology textbook for their studies (n=24).

• 55.0% of students told us OpenStax College textbooks were required reading and they did not purchase any other textbooks for their course (n=22)

Educator Overview

• 92.6% of respondents have adapted OER to fit their needs• Just under 50% of respondents have created OER for study or teaching

(48.1%, n=39) • The biggest challenge for respondents when using OER was finding

resources of a sufficiently high quality (67.5%, n=54)*• Top three types of OER used for teaching/training by respondents: open

textbooks (98.9%, n=81), videos (78.0%, n=64) and images (72.0%, n=59) • Almost 60% of respondents think their institution benefits financially by using

OER (59.3%, n=48)

Educator Overview

• 34.6% of respondents first became aware of OpenStax College Textbooks via an internet search (n=28).

• Over a quarter of respondents became aware of OSC via a colleague or other recommendation (25.9%, n=21)

• 61.0% of respondents currently use the College Physics textbook with their students (n=50).

• Over 80% of respondents teach Science (82.7%, n=67) with 77.8% using OER in this area (n=63) and over a third creating OER in this subject (34.6%, n=28)

Student Perspectives

In which of these ways (if any) has your use of Open Educational Resources made an impact on your formal

studies? (Check all that apply)

"Using Open Educational Resources (e.g. OpenStax College textbooks) has led to my...”

Increased interest in the subjects taught (69.0%, n=29)Increased satisfaction with the learning experience (66.7%, n=28)Grades improving (47.6%, n=20)Increased participation in classroom discussions (45.2%, n=19)Having increased independence and self-reliance (45.2%, n=19)Increased engagement with lesson content (45.2%, n=19)Increased experimentation with new ways of learning (45.2%, n=19)Gaining confidence (38.1%, n=16) Increased enthusiasm for future study (38.1%, n=16) Increased collaboration with my peers (35.7%, n=15) Being more likely to complete my course of study (33.3%, n=14) Becoming interested in a wider range of subjects than before I used these resources (31.0%, n=13)

In what ways, if any, has using OpenStax College textbooks impacted on your studies?

Range of responses, including comments on:

40.6% Better Study Experience (n=13)

31.3% Accessibility of Resources (n=10)

12.5% Cost savings (n=4)

9.4% Improved Grades (n=3)

9.4% Less Weight to Carry (n=3)

In what ways, if any, has using OpenStax College textbooks impacted on your studies?

“Made learning easier and more accessible, helped and broadened my understanding and

learning, made me WANT to study, helped me become interested in Biology because of the

easy layout and presentation.”

“Saved my back, I already have so many other books in my backpack and using my computer for more info saved additional

stress that would have been on my back.”

“Increased my Ap Biology grade

dramatically.”

(Earlier this respondent reported: “I

went from being horrible in AP

Biology to actually reading these

and went from a D 66% up to a

A 90% So far.”)

“Open Stax textbooks offer me

peace of mind and permit me to

study in a more relaxed way. I am

grateful not to have to come up with

hundreds of dollars per class for

books.

“It has allowed for

me to develop

knowledge easily in

areas that I thought

would be difficult to

learn in due to the

inability to buy an in-

depth textbook.”

Educator Perspectives

Based on your experiences as a teacher, to what extent do you agree with the following statements?

% of educators who strongly agree or agree with the following statements:

"Use of Open Educational Resources (e.g. OpenStax College textbooks) in the classroom...”

Allows me to better accommodate diverse learners’ needs (67.9%, n=53) Increases learners’ satisfaction with the learning experience (65.8%, n=52)Increases learners’ experimentation with new ways of learning (64.1%, n=50)Increases learners’ engagement with lesson content (60.3%, n=47)Increases learners’ interest in the subjects taught (55.7%, n=44)Develops learners’ increased independence and self-reliance (55.7%, n=44)Increases learners’ participation in class discussions (49.4%, n=39)Leads to improved students’ grades (47.4%, n=37)Increases learners’ enthusiasm for future study (45.6%, n=36) Builds learners’ confidence (45.5%, n=35)Increases collaboration and/or peer-support among learners (44.2%, n=34)Leads to learners becoming interested in a wider range of subjects than before they used OER (41.0%, n=32)

In what ways, if any, has using OpenStax College textbooks impacted on your own teaching practice?

“I have become more intentional about tailoring my course to my students, having been at least

somewhat released from the constraints imposed by the commercial publishing industry”

“I have always believed in using a lot of

additional resources. OpenStax works

nicely in this regard. The supplementals are not fancy but we have made

additional testbanks and other free resources available to all our faculty in

a collective approach to compile very nice additional resources.”

“I am able to think a little more creatively of what new

experiences I can design for my students using

resources in the text or other online resources. I think

the text is a great foundation, but I don’t feel

constrained to shy away from employing other

teaching tools for fear they might ask: “Well, if you

were going to have us watch this video or do this

online tutorial – why did we have to pay $250 for the

textbook?”

“I create more collections of images to build presentation,

rather than going straight from the book.”

In what ways, if any, has using OpenStax College textbooks impacted on your students?

“They are able to access the textbook and start doing homework immediately rather than being

delayed until weeks after the start of the course due to lack of finances.”

“Ease of use for students and

parents. The 'note-taking" type of

students can print pages, and when

I find students have a specific

difficulty I can even print a page to

use in class. Faster students can

read on devices at any time - they

enjoy viewing the book from

smartphones etc.”

“They have the book as a resource

after leaving class. With another

text they would most likely sell it

back. Also sharing content with

others outside of class.”

“Increased reading of the

textbook, and a portable

textbook that can go with them

for review. Unlike other e-

textbooks, this one can stay

with them.”

“…the students are sharing notes and

annotations online.”

“The resource gave them a sound understanding of the topic and they were able to do self-directed study from it. The result is that seniors have excelled in their exams.”

In what ways, if any, has using OpenStax College textbooks impacted on your students?

“more student engagement in lab activities and lecture presentation”

“Free and available online (while they

work, or are at home) make OpenStax

texts a great excuse-defying tool. In a

positive light- students have access to the

text wherever and whenever they

need/want it.”

“…Students love having multiple options

for textbook format (epub, iBook, pdf, or

hardcopy). Enrollment in my class has

increased by 25% - can't say for sure it

is due to the textbook, but it certainly

has helped. …”

“It saves all 800 students per

year about $120. One student

told me that she had to work 15

hours for each book that she

has to buy. That really put it in

perspective.”

“Students seem somewhat more engaged.”

“I think that it is highly beneficial to have a

brand new text to use, I would have been

forced through budgetary constraints to

purchase other texts which are 5-10 years old..”

As a result of using OpenStax College textbooks…

• 96.3% of educators who use OSC textbooks are more likely to recommend OpenStax College textbooks to fellow educators/teachers (n=77)

• 81.0% of respondents who use OSC textbooks are more likely to discuss using OpenStax College textbooks with their institution’s administrators (n=64)

• Over three quarters of respondents who use OSC textbooks are more likely to use OER for teaching (77.9%, n=60)

Financial Savings

Student Financial Savings

“Yes” 77.3%

“No” 15.9%

“Don’t Know” 6.8%

Do you think you have saved money by using OpenStax College text-

books?

“Yes”89.0%

“No”8.5%

“Don’t Know”2.4%

Do you believe that your students have saved money by using OER?

By using OpenStax College textbooks each student surveyed saved on average ca. $155 (n=19)

Summary

Summary of ResultsEfficacy: Are they Learning?

• Over 60% of both students and educators report increased satisfaction with their own or their students’ learning experience as a result of using OER/OSC textbooks

• Almost 70% of students surveyed told us that they thought use of OER/OSC textbooks had led to an increased interest in the subjects taught

• Over 45% of students thought that their grades had improved due to using OER/OSC textbooks

• Emerging evidence for different ways in which “open” is making a specific difference: students can both access materials earlier (e.g. pre-course/pre-payment of financial aid) but can also continue to access/use resources post-course (don’t have to sell textbook/limited access). Positive impact on learning?

Next Steps

Next Steps

• Further analyses of data: e.g. comparison of adopters with newsletter respondents (educators), non-users/users of OSC textbooks, students studying in different contexts, different OSC textbooks etc.

• Comparison of data with other open textbook surveys

• Disseminating results (Impact Map inc. anonymised data)

• Future/current work focused on building case studies and comparative impact data

• We need you! Looking for YOUR best examples of impact...

• Help us build the most comprehensive picture of the impact of OER by contributing your evidence to the Impact Map

Thanks for listening!oerresearchhub.orgchaos.open.ac.ukTwitter: @OER_Hub @BeckPitt

in service of The Open University