IEEE TRANSACTION ON EDUCATION, VOL44, NO.3, AUGUST 2001

Web-Based Peer Review: The Learner as both Adapter and

Reviewer王舜楷

MINELab2011 / 08 / 26

Outline

• Introduction• Theories• Research Methodology• Research Questions• Result and Discussion• Conclusion and Implication

Introduction

• In recent year, systems and learning theories have been developed for web-based learning in higher education, while the effectiveness of those implements have been empirically evaluated as well.

• Higher level learning such as cognitive monitoring or critical thinking has received relatively little attention.

Introduction (cont.)

• Researcher concur that cooperative learning is the major approach when learning and teaching learning theories are distributed by WWW.– Utilizes the web as a media for

information distribution and a channel for group interaction.

– How to create the conductive environments to promote effective group interaction?

Introduction (cont.)

• Web-base learning strategy, Peer review– Values exchanging critical feedback among peers

and modifying works according to peer feedback.– An authentic learning environments can hopefully

be fostered in which students actively construct knowledge similar to the manner in which scientist do.

– Students take the role of both researcher and reviewer.

– Teacher function like a journal editor – teacher evaluate each student’s assignment after peer review process.

Theories

• Constructivism– Social constructivism

• Peer Review• Metacognition

Theories (cont.)

• Constructivism– The construction of new knowledge

starts from one’s observation of events through past experiences one already has.

– Learning new things involves changing the meaning of one’s previous experience – learning means to construct or modify one’s inner knowledge structure.

Theories (cont.)

– Learning should be evaluated by revealing a learner’s inner knowledge structure rather than measuring behavior performance.

– Therefore, web learning strategies in line with constructivism must focus on how computer networks support active knowledge construction of learners.

– Social constructivism.

Theories (cont.)

– Social constructivism claims that although an individual is responsible for developing one’s own knowledge structure, learning a concept requires exchanging , sharing, and negotiation, as well as occasionally drawing on the expertise of more knowledgeable individuals.

– Learning involves both personal inner process and social aspect, and social constructivism uniquely emphasizes the social aspect.

– This theory draws attention to the social context that individuals share or question other’s ideas to construct and modify their own knowledge.

Theories (cont.)

– Following such a trend, the goal of teaching is to achieve shared meaning and active participation of all students.

– To create an authentic environment, a teacher must motivate students’ active knowledge construction. Moreover, teacher must be available to assist students when needed.

– Teachers are more like knowledge “old-timer” who facilitate new comer (student) in appropriate community-specific practices and resource.

Theories (cont.)

• Peer Review

– Peer review effectively allows academic societies to construct knowledge through social sharing and competition.

Individual work in HTML format

Post at a demo area andassign several peer reviewers

for each student

Reviewers rate and commenton other’s homework

Teacher grade homeworkbased on the evaluationof the peer reviewers

feedback

After 3 rounds

Theories (cont.)

– Some studies may only ask students to evaluate and comment on peer’s work in a summative way, and that makes less peer pressure.

– Some problems• Students may underrate their peers or

overrate them.• Agreement between teacher evaluation and

peer evaluation was not highly enough to demonstrate the validity of peer review.

Theories (cont.)

• Matacognition– Metacognition has two separate but relate

d aspects: knowledge and regulation of an individual’s cognition.• The knowledge aspect refers to knowing about

one’s own cognition status, knowing about tasks, and knowing about learning strategies.

• The regulation aspect in contrast refers to the control one’s cognition process.

Theories (cont.)

– What do students do in the process?• In completing one’s own project, a student may

plan the contents and procedure to be shown, then execute the plan.

• In reviewing peer homework, one must read, compare, or question ideas, suggest modification, or even reflect how well one’s own work is compared with others.

• While processing these cognitive functions, one has to monitor the adequacy of cognitive function adopted. If not adequate, or ineffective, one then must regulate cognitive functions accordingly.

Theories (cont.)

• Many students, particularly those with low motivation and achievement, are unwilling to do mindful work.

• Learners in peer reviewer process receive comments from peers, but not directly from teacher as in the traditional process.

• Therefore, peer pressure, as a motivate factor, may push students to perform higher level cognitive function.

Research Methodology

• Participants and Contents• Measurements

– Achievement– Perceived Satisfactory– Metacognition

• Web-based Peer-Review System

Research Methodology (cont.)• Measurement: Achievement

– Three indexes: assignment, review, and final exam.• The assignment score, rated by the teacher based

on peer’s evaluations, was a combination of assignment performance in 3 rounds of peer review.

• The review score, an indicator of each reviewer comment quality, was rated by both the teacher and TAs.

• The final score , based on multiple choice and essays in a standard paper-pencil test, was a summative evaluation of the students’ achievement.

Research Methodology (cont.)• Measurement: Perceived Satisfactory

– Measured by a questionnaire containing ten questions about the peer review as well as the adequacy of review criteria.

– Students were asked to rate their satisfactory on a five-point Likert scale.• About effectiveness, motivation, and review cri

teria.– An open-ended attached to each statement

s to elicit free opinion or modification suggestions for peer review and WPR system.

Research Methodology (cont.)• Measurement: Metacognition

– The authors looked for qualitative evidence of several cognitive and metacognitive strategies.• Critical thinking• Planning• Monitoring• Regulation

– A review comment containing keywords from any of these four kinds of higher level thinking was coded into an adequate category.

Research Methodology (cont.)• Web-base Peer-Review System: WPR

– As an information distribution channel and management center for assignment submission and peer review.

– Sa s media for peer interaction and knowledge construction.

– As s storage center for knowledge construction procedures.

– Teacher and students can post information in a BBS designed for the course to express their opinion about the course and system.

Research Methodology (cont.)

– Assign homework in HTML format and CGI program for user to upload homework• Ensure easy access to all uploaded

assignments through Internet.

– Blind evaluation.• Ensure fairness and willingness to critique.

– Duplicate each assignment to another directory before evaluation of each round.• Keep different assignment versions in various

rounds intact for future analysis.

Research Question

1. Do computer science majors learn effectively through WPR?

2. Do students perceive to benefit from peer review and prefer using it? Do students express negative attitude toward peer review?

3. Do students display metacognition skills during peer review?

4. Does the effective author or the critical thinking reviewer perform better in the final examination? Is an effective author always a good reviewer?

Result and Discussion

• Q1– The assignment score was achieved by

a student under peer review, while final score was gained by a student alone.

– Comparing the two scores, students performed better with peer review.

– However, this study has no control group, thereby making the causal effect of peer review uncertain.

Result and Discussion (cont.)• Q2

– 64.71% of the participants viewed peer review as effective and that they benefited from using this learning strategy.

– Through open-ended questions, students reported that the benefit of peer review came from reading many peer’s work.

– Many students mentioned that they compared their own work with peers’ work and were more aware of their advantages and weaknesses than when in conventional teacher evaluation situation.

Result and Discussion (cont.)

– 67.86% of the participants reported that they prefer using peer review. However, about 9% of the participants questioned the fairness of peer review and suggested decreasing the weight of peer review and increasing the weight or teacher evaluation in the total score of the course.

– Regarding the review criteria, many students founded that a total score with a global comment for an entire assignment might not offer sufficient information for further revision.

Result and Discussion (cont.)• Q3

– 77% of reviewers displayed all four skills of higher level thinking.

– However, critical thinking and monitoring were more frequently used than planning and regulation.

Result and Discussion (cont.)• Q4

– Result showed that multiple correlation among assignment, review, and final exam were significant.

Final examination

Assignment

ReviewCorrelation betweenAssignment and Review

Correlation between Assignment and Final exam

Correlation between Review and Final exam

Multiple Correlation

Result and Discussion (cont.)

– In summary, the above correlation analysis reveals that merely being an effective reviewer may not gain a high grade in the final course evaluation.

– An effective reviewer must also be a quality author to ensure a high final score.

– However, effective authors are capable of making high quality comments in peer review.

Result and Discussion (cont.)

Assignment score

High Low

ReviewScore

HighStrategic Adapter

Effective Reviewer

LowSelf-centered

LearnersFailure

Conclusions and Implications• While this study did not use an

experimental design, these finding are preliminary in nature.

• This study has demonstrated that peer review is an effective web-based learning strategy. During the peer review process, students displayed higher level thinking.

• Students also reported they were motivated to learn more through peer review process.

Conclusions and Implications (cont.)• Merely being an effective reviewer or an effective

authors may not succeed in a peer review environment. The one who excels the most is strategy adapter who can effectively construct assignments, adjust to peers, comments and, at the same time, be a critical reviewer.

• Similar to self-regulated learner, who can set personal education goal and then generate thoughts, feelings, and actions to attain them.

• It is believed that the peer review strategy is also effective for undergraduate students in related fields.

• Based on the above finding, the authors conclude that the WPR system creates an authentic environments, as suggested by social constructivism.

Conclusions and Implications (cont.)• Future Study

– Examine the effectiveness of WPR in different situation.

– The author encourage using experimental designs to confirm the causal effect of WPR.

– Merely providing a learning-assisted system may not necessarily promote active knowledge construction. An effective learning strategy supported by the system is of priority concern.

Conclusions and Implications (cont.)

– Further study should also analyze the reasons some students cannot adapt well in peer review (Failures in P.29 Table).

– Teachers should not only closely monitor self-centered learners and effectiveness reviewers, but also analyze what cognitive strategies they lack.

– In doing so, all students can eventually be equipped to become strategic adapters, thus equipping them for challenge in the workplace as well.

Thank you for your attention