By David W. Reedy and James W. Pagel Associate Professors School of Science and Engineering

A Five-year project CALL Project Leads to the Analysis of Motivation with an Eye on Curriculum Reform

By

David W. Reedy and James W. PagelAssociate Professors

School of Science and EngineeringAoyama Gakuin University

Tokyo, Japan

14th International CALL Conference08/20/2010

This presentation deals with:

Our previous study on implementing an e-learning component in all English classes offered in our department;

The current investigation of whether entrance routes have any affect on performance on post tests two years after university entrance, and whether there is any correlation between completion of the e-learning project and an increase in test scores.

A basic explanation may be necessary.

This research is the culmination of the authors’ efforts to implement and e-learning component in all English classes in their department to comply with MEXT mandates.

It is our goal to improve the department’s English curriculum.

The Ministry of Education, Culture, Sports, Science and Technology (MEXT) has set various targets. In 2001 the Rainbow plan was announced: Smaller classes, easier-to-understand lessons and Integration of IT into the classrooms In 2004 the agency stated its goal of having all

eligible High School graduates be accepted to university

In 2005 in its initiatives for higher education, called for a fierce competition among private universities, and for globalization, which meant:

1. Accepting more students from overseas;

2. Sending more students abroad;

3. Opening satellite campuses;

4. Offering joint degree projects with other universities.

Globalization is the key word, but…Many universities* reported that they lacked the money the staffing the infrastructure

to implement e-learning on a large scale basis

* 60 percent answering the survey by the National Institute of Multimedia Education

Source: Japan Times “E-Learning: Making Inroads into Nation’s Colleges” May 7, 2007

Japanese students are not so interested in going overseas.Students are content with their

situation “at home.” Internet research is sufficient.(And companies can no longer afford to send its workers overseas as in the past.)

Students fear they lack the ability to succeed in foreign countries.

Reverse Japan’s Insularity. Japan Times April 8, 2010 Forty-nine percent of new hires will not go abroad. Jiji Press August 13, 2010

This is the start of the study.Integrate commercial software in

all departmental English classes.

But only 36.3% of Japanese universities had offered e-learning classes as of 2006.

Purchase of commercial software was “one-time only.”

Analyze the merits of such software.

In short, we succeeded.

We needed to know how students reacted to such software.

Our survey results were inconclusive.

938

Why did they stop?

9a. Why did you give up? 2006 2007Hard to maintain motivation 90 (40%) 91 (54%)

Takes too much time 72 (32%) 33 (20%)

“Typically, drop-out rates are reported to be 80 percent or more in e-learning environments where little or no classroom support is available.

“ Evolution of CALL” by Lance Knowleshttp://www.dyned.com/pdf/Teacher-Guides/TGTHEORY.PDF

Here is some more information.1. Why didn’t you try the material? 2006 2007Too busy 454 (61%) 228

(65%)

Bonus points 110 (15%) 25 　(7%)

too low Dislike English 68 (9%) 34

(9%)

Our target became how to identify students who completed the e-learning component and their motivation.

Break down the students by department. (2010)

Seek other ways of finding correlations.

Reinvent the student survey.

We broke down students by department.

N=564

So you are now up to speed.What is next?

We could analyze completion of the e-learning component other ways. And why…(problem: survey answers were anonymous.)

We could study how students are admitted to the university to better understand motivational factors in the study of English.

There are ten ways to get accepted.DepartmentalExam Day OnePassing on firstcut-off 224 (39.7%)

DepartmentalExam Day OnePassing onsecond cut-off 45 (8%)

DepartmentalExam Day TwoPassing on firstcut-off100(17.7%)

DepartmentalExam Day TwoPassing onsecond cut-off4 (0.7%)

National Exam First Term20 (3.6%)

National Exam Second Term +Departmental Exam21 (3.7%)

Internal High School Evaluation30 (5.3%)

Recommendation120 (21.3%)

International Students ExamForeign Students 2 (0.4%)

International Student ExamStudents attending internationalhigh school1 (0.2%)

We considered success on the basis the following.

TOEFL ITP Pretest results averages Posttest results averages Best score change averages Percentage of students whose scores increased Percentage of students whose scores decreased e-learning Percentage of freshmen who completed the requirement Percentage of freshmen who completed the bonus

component Percentage of sophomores who completed the

requirement Percentage of sophomores who completed the bonus

component

TOEFL ITP pretest scores University Entrance Mean SD

1. Departmental Exam Day One: Passing on first cut-off

422.4 32.62

2. Departmental Exam Day One: Passing on second cut-off

411.4 30.18

3. Departmental Exam Day Two: Passing on first cut-off

416 26.49

4. Departmental Exam Day Two: Passing on second cut-off

394 17.1

5. National Exam 442.6 30.24

6. National Exam + Departmental Exam 417.2 30.61

7. Internal High School Evaluation 405.3 13.49

8. Recommendation 398.7 29.32

9. International Students Exam: Foreign Students

383.5 3.5

10. International Student Exam: Japanese students attending foreign high school

540 0

Pretest and Posttest scores

University EntrancePretest Mean

SD Posttest Mean

SD

1. Departmental Exam Day OnePassing on first cut-off

422.4 32.62 410.8 41.8

2. Departmental Exam Day OnePassing on second cut-off

411.4 30.18 392.7 45.1

3. Departmental Exam Day TwoPassing on first cut-off

416 26.49 388.6 38.8

4. Departmental Exam Day TwoPassing on second cut-off

394 17.1 389 13.4

5. National Exam 442.6 30.24 428.1 44.86. National Exam + Departmental

Exam417.2 30.61 405.4 39.9

7. Internal High School Evaluation

405.3 13.49 401.4 35.4

8. Recommendation 398.7 29.32 390 36.59. International Students ExamForeign Students

383.5 3.5 XX XX

10. International Student ExamJapanese students attending foreign

high school

540 0 590 0

Total 412.62 32.65 401.21 42.03

The Spearman rank-order correlation coefficient calculates to .82.

University Entrance Pretest Ranking Post test Ranking

10.International Student ExamJapanese students attending foreign high school

1 (540) 1 (590)

5. National Exam 2 (442.6) 2 (428.1)


3 (422.4) 3 (410.8)

6. National Exam + Departmental Exam

4 (417.2) 4 (405.4)


5 (416) 9 (388.6)


6 (411.4) 6 (392.7)

7. Internal High School Evaluation 7 (405.3) 5 (401.4)

8. Recommendation 8 (398.7) 7 (390)


9 (394) 8 (389)

9. International Students ExamForeign Students

10 (383.5) XX XX

How did scores change? Entrance Route Average

TOEFLChange

ScoredHigher

ScoredLower

No Change

StudentsAbsent


-10(-2.4%)

64(39%)

99 (60.4%)

1 (0.6%)

59(26.8%)


-18.7(-4.6%)

12(36.4%)

21(58.3%) 0 12

(26.7%)3. Departmental Exam Day TwoPassing on first cut-off

-27.4(-6.6%)

14(23.3%)

46(76.7%)

2(3.3%)

38(38%)


-5(-1.3%)

1(25%)

2(50%)

1(25%)

0(0%)

5. National Exam -14.5(-3.3%)

3(18.6%)

12(75%)

1(6.3%)

4(20%)


-11.8(-2.8%)

6(35.3%)

11(64.7%) 0 4

(19.1%)7. Internal High School

Evaluation-3.9

(-1%)6

(28.6%)12

(57.1%)3

(14.3%)9

(30%)8. Recommendation -12.4

(-3.1%)39

(37.9%)58

(56.3%)6

(5.8%)17

(14.2%)9. International Students ExamForeign Students

XX XX XX XX 2(100%)

10. International Student ExamJapanese students attending

foreign high school

+50(+9.3%)

1(100%)

0 0 0

Total -11.4(-2.76%)

146(34.68%)

261(63%)

14(3.33%)

145(25.62%)

TOEFL achievements at a glance.Entrance Route pretest post test up down change


3 3 2 6 4


6 6 4 5 8


5 9 8 9 9


9 8 7 2 3

5. National Exam 2 2 9 8 7


4 4 5 7 5

7. Internal High School Evaluation 7 5 6 4 2

8. Recommendation 8 7 3 3 6


10 XX XX XX XX


high school

1 1 1 1 1

Give your eyes a little break!

Are you now ready for more?

I am David & on vacation

Our e-learning program at a glance.

1st term 2nd term

Freshmen Requirement 6 units

Bonus Component30 units

Sophomores Requirement 6 units

Bonus Component30 units

They always do better as freshmen.Entrance Route 1st

term req

1st term bonus

2nd term req

2nd term bonus

1. Departmental Exam Day OnePassing on first cut-off 89.73 33.33 75.89 13.392. Departmental Exam Day OnePassing on second cut-off 91.11 34.15 71.11 22.223. Departmental Exam Day TwoPassing on first cut-off 81 44.44 69 104. Departmental Exam Day TwoPassing on second cut-off 75 66.67 75 505. National Exam 78.95 26.67 63.16 10.536. National Exam +

Departmental Exam 90.48 26.32 61.90 4.76

7. Internal High School Evaluation 93.33 60.71 86.67 23.33

8. Recommendation 98.33 52.54 86.67 30.839. International Students ExamForeign Students 0 0 0 010. International Student ExamJapanese students attending

foreign high school100 0 100 100

Total 89.22 35.34 73.14 17.67

Let’s focus on E-learning achievements.

Entrance Route 1st yr req

1st yr bonus

2nd yr req

2nd yr bonus


6 6 4 6


4 5 6 5


7 4 7 8


9 1 5 2

5. National Exam 8 7 8 76. National Exam + Departmental Exam 5 8 9 9

7. Internal High School Evaluation 3 2 2 4

8. Recommendation 2 3 2 3


10 9 10 10


high school

1 9 1 1

Let’s tally up the two groups.Entrance Route TOEFL

total ranke-learning total rank


2 6


7 5


9 7


7 4

5. National Exam 5 8

6. National Exam + Departmental Exam 4 9

7. Internal High School Evaluation 3 3

8. Recommendation 5 2

9. International Students ExamForeign Students 10. International Student ExamJapanese students attending foreign high

school

1 1

We drew the following conclusions.Success can’t be determined by

comparing entrance route and TOEFL scores. The Spearman rank-order correlation coefficient of TOEFL and e-learning achievements calculates to .44.

Students entering via the internal high school evaluation system were most consistent (ranked 3rd in both categories).

More data analysis is necessary.

Our current investigations have led us to consider:Limiting comparison to top and bottom

scoring students;Looking at individual departments and

their students’ pretest and posttest scores vs e-learning completion rates, considering the types of classes required);

Revising the student survey tool to include more questions that would address motivation.

References not listed above Ministry of Education, Culture, Sports, Science and Technology.

(2001). The education reform plan for the 21st century – the Rainbow Plan. Retrieved from http://www.mext.go.jp/english/topics/21plan/010301.htm

Ministry of Education, Culture, Sports, Science and Technology.

(2003). Regarding the establishment of an action plan to cultivate ‘Japanese with English abilities’. Retrieved from http://www.mext.go.jp/english/topics/03072801.htm


(2008). Recent MEXT’s Initiatives for Higher Education. Retrieved from http://www.jpf.go.jp/cgp/exchange/event/pdf/overview_ota.pdf


(2009). Launching the Project for Establishing Core Universities for Internationalization (Global 30) Retrieved from http://www.mext.go.jp/english/news/1283454.htm

Thank you for your attention!

Any questions?

James W. Pagel David W. Reedy [email protected] [email protected]

14th International CALL Conference08/20/2010

Documents

By David W. Reedy and James W. Pagel Associate Professors School of Science and Engineering