Morphing Rubrics to Adapt to Developmental Student NeedsAja Henriquez, MFA, EdD Student CSUSBEnglish Instructor:Crafton Hills College, California Baptist University

From Collins English Dictionary

What is a Rubric? Definition of rubric (ˈruːbrɪk)

1.  

a title, heading, or initial letter in a book, manuscript, or section of a legal code, esp one printed or painted in red ink or in some similarly distinguishing manner

2.  

a set of rules of conduct or procedure

3.  

a set of directions for the conduct of Christian church services, often printed in red in a prayer book or missal

4.  

instructions to a candidate at the head of the examination paper

5.  

an obsolete name for red ochre

6.  

written, printed, or marked in red

Why Rubrics?

School Specific IssuesApproximately 31% of incoming students

transfer into the developmental level (English 015) at CHC, while 36% transfer into a lower, remedial level course (914): Around 33% of students test into college level English courses (Research Briefs, 2011).

Approximately 52% of students who pass the “Preparation for College Writing” course (English 015) persist in the college and successfully complete a college level writing course (Student Equity Data, 2011).

What We Can ControlThe success rates tell us that there is

something not transferring in our courses.

Many things we cannot control: student readiness/commitment, family problems, financial problems, etc.

We can address our rubrics and how we communicate with the students.

As we know from research, the students may simply misunderstand what we say/write when we grade.

What is the Purpose of a Rubric?We use it to grade for a few

reasons:◦To assess specific items in a given

assignment◦To help ensure or increase uniformity

of assessmentIt is a tool for instructors to

communicate with students. Students must be able to

understand the communication.

Language Disconnect We may be taking for granted student understanding of

our rubrics. Developmental courses often act as an introduction

into the academic discourse community (Bizzell, 1982): “students from different social classes come to

school with different abilities to deal with academic discourse: middle-class students are better suited by their socialization in language use to deal with academic discourse's relative formality and abstraction than working-class students are. This unequal removal from academic language is, of course, exacerbated for students whose home language does not resemble the so called standard English” (192).

Student Perspective

VS

“It’s like they [the affluent students] had their own classes, and we [the students from the poor/working class] had our own classes. We were, like, segregated in the same school. “

--Prep for College Writing Student

Demographics of Basic Skills Students“Crafton Hills College, then,

serves a community where the constituents must commute to work or school, where the poverty and unemployment rates exceed the state average, and where the income is below the national average.”

From Demographics, CHC webpage: http://www.craftonhills.edu/About_CHC/Demographics.aspx

We Should Keep in Mind:Students may not understand the

language we use on rubricsWe must ask ourselves if we

have written the rubric for our understanding or student understanding.

Example: Unexamined Rubric

Excellent Good Adequate Poor Needs WorkControlling Idea:

Organization:

Development:

Audience:

Comments:

A Tool That Doesn’t WorkDoes a developmental student

know what these terms mean when it comes to writing?

If the students don’t understand what the rubric means, then they can’t use it to improve.

How can we make this more understandable?

Example: Revised Rubric

Other ConcernsEven with some clearer criteria,

there are still some issues with this rubric, which are less simplistic than just the language used.

Validity-does this measure what we want it to measure?

Reliability-are the scores consistent?

Messick (1995)

Messick (1995)

ValidityContent Validity: to ensure we are

measuring what students should know and not some other thing (personal preferences), we must match each item on the rubric to a course objective.

Substantive Validity: we should make sure that the items on the rubric connect to different types of cognitive processes and are of varying difficulty

How to Ensure Content Validity Review the items on your rubric

and make sure they match to the course objectives along with any departmental norms.

English departments often have norming sessions for grading, which these instructors must take into consideration when evaluating the content of their rubrics.

EXAMPLE partial rationalization of content validity

Some things that make it into our grading that should not be there: stapling, niceness of binder or folder, whether it was accompanied with a latte, etc.

Item one corresponds with objectives 4, 5, 6, 11, as well as the grading criteria for writing (demonstrates skillful use of vocabulary and syntax; is generally free from errors in mechanics, usage, and sentence structure).

Item two corresponds with objectives 7, 8, 9, as well as the grading criteria for writing (focuses clearly on the topic and responds effectively to all aspects of the assignment; explores the issues thoughtfully and in depth; is coherently and logically organized with a thesis statement supported by apt reasons and specific, detailed examples).

Substantive Validity

For explanation of Bloom's Taxonomy, watch http://youtu.be/qjhKmhKjzsQ

Ensuring Substantive ValidityFirst of all, we should examine

the items in our rubrics to ensure there is a spread of difficulty.

By doing this, we are able to measure the ability of students who are at different levels ability (low, mid, and high).

Substantive ValidityPERSONS - MAP - ITEMS <more>|<rare> 2 k + | T| | | | m | | | | | j | | | S| 1 + n | | b | f | | i | | | e | h | M|T Q3 o p | |S | 0 c g +M Q2 Q4 | Q1 |S | Q5 |T | | a | S| | | | | | | -1 + | | l | | T| | d | | | | | | | | -2 + <less>|<frequ>

Items do not match the student ability well.

It is difficult to establish which items should be more difficult, since each individual will have personal difficulties.

There should be a greater spread of difficulty.

Rasch ModelThe previous slide showed an

output table from Winsteps, which is based on the Rasch mathematical model.

We don’t all have this program or the ability to use it.

There is a simpler way to examine our student scores using Excel, which most of us already have.

Steps to Examining ScoresOpen an Excel pageInput the student scores for each

item and their total score.You do not need to input student

names. Simply using letters will work.

Using Excel, array the scores from highest to lowest.

Examine the scores.Step by step instructions linked on http://www.youtube.com/watch?v=nDez8VmlN9A

Classical AnalysisStudent #

Writing Conventions

Integration of Research Format

Content/ Development Organization Total

Points Possible 12 16 12 48 12 100 k 12 16 12 48 12 100 m 12 16 12 46 12 98 j 10 16 12 46 12 96 n 12 12 12 48 9 93 b 9 16 10 44 12 91 f 6 14 10 48 12 90 i 9 12 9 46 12 88 e 10 12 9 44 10 85 h 9 16 9 40 9 83 o 9 12 9 40 9 79 p 9 12 9 40 9 79 c 12 4 3 44 12 75 g 12 8 6 40 9 75 a 9 4 3 40 9 65 l 6 0 4 40 6 56 d 9 0 3 36 6 54

155 170 132 690 160

item variance 3.59 30.36 10.69 12.98 4.13

Pt. B Cor. 0.41 0.93 0.92 0.84 0.82

Item one does not have consistent scores.

In order to fix this, I would expand this item, Writing Conventions, into its component parts.

Adding more items will help pinpoint the exact issues with writing conventions that give students trouble, which will in turn help me differentiate instruction based on student need.

Fix Through Metamorphosis

Morphed Rubric, Item OnePerformance

Levels Dimensions

 Exemplary (4)

 Accomplished (3)

 Developing (2)

 Beginning (1)

 Score

Spelling 

There are no spelling errors.

Spelling errors are minimal (1/page)

Spelling errors are a problem (2-3/page) 

Spelling errors are a problem (4+/page)

/12

Punctuation There are no punctuation or capitalization errors

Punctuation errors are minimal (1 per page)

Punctuation errors are a problem (2-3 per page)

Punctuation errors are excessive (4+ per page)

/12

Fragments There are no fragments in the essay.

Fragments are minimal (1 total)

Fragments are a problem (2 total)

Fragments are excessive (3 or more total)

/12

Run-ons There are no run-ons in the essay.

Run-ons are minimal (1 total)

Run-ons are a problem (2 total)

Run-ons are excessive (3 or more total)

/12

Interpreting the MorphBecause item one was so problematic, I

can interpret that it is not very clear what the students can do to improve, even with explicit in-text notes. (As we may know)

I morphed it by expanding so that both the students and I could understand what wasn’t working in their assignments. ◦They can work on spelling instead of run-ons◦I can work on student weaknesses with

instruction

The Rasch analysis with Winsteps suggests that most of the grade scale is not necessary.

Grading on a 100 point scale suggest the ability to differentiate an essay in 100 ways (Dr. Jesunathadas).

As the Revised Rubric suggests, I would use a 1-4 point scale for each item and then report the final score as a percentage for student understanding.

The changes to the grading scale will help keep the scoring effective (no more half scores that don’t differentiate well).

Hidden Issues with Rubrics

Item 2ITEM NUMBER: 2 Integration of Research

CATEGORY PROBABILITIES: MODES - Structure measures at intersections

P -+---------+---------+---------+---------+---------+---------+-

R 1.0 +0000000000000 6666666666+

O | 000 6666 |

B | 00 66 |

A | 0 6 |

B .8 + 0 6 +

I | 0 4 6 |

L | 0 4 4 6 |

I | 0 4 4 22 6 |

T .6 + 0 4 4 2 2 6 +

Y | 0 4 4 2 2 6 |

.5 + * 4 2 2 6 +

O | 40 42 26 |

F .4 + 4 0 * 62 +

| 4 0 * 6 2 |

R | 4 0 24 6 2 |

E | 4 0 2 4 6 2 |

S .2 + 4 0 *8* 6 2 +

P | 4 0 * * 6 444 2 |

O | 44 8*2 4**44 444* |

N | 444 882200 **4488 ***444 |

S .0 +*************************************************************+

E -+---------+---------+---------+---------+---------+---------+-

-3 -2 -1 0 1 2 3

PERSON [MINUS] ITEM MEASURE

Revised RubricAssignment Rubric: REVISED ESSAY RUBRIC

Performance Levels

Dimensions

Exemplary (4)

Accomplished (3)

Developing (2)

Beginning (1)

Score

Spelling

There are no spelling errors.

Spelling errors are minimal (1/page)

Spelling errors are a problem (2-3/page)

Spelling errors are a problem (4+/page)

/4

Punctuation There are no punctuation or capitalization errors

Punctuation errors are minimal (1 per page)

Punctuation errors are a problem (2-3 per page)

Punctuation errors are excessive (4+ per page)

/4

Fragments There are no fragments in the essay.

Fragments are minimal (1 total)

Fragments are a problem (2 total)

Fragments are excessive (3 or more total)

/4

Run-ons There are no run-ons in the essay.

Run-ons are minimal (1 total)

Run-ons are a problem (2 total)

Run-ons are excessive (3 or more total)

/4

Integration of research

Research is integrated to illustrate and support main points with specific details. At least 3 sources used.

Research is referenced, but specific details are sparse. Only 2 sources used.

Research is referenced, but specific details are not used. 1 source used.

Research is not discussed or referenced, and print out not attached. No sources used

/4

Proper MLA format

Proper margins, font, spacing, header, etc. are used

Proper formatting is followed except for 1-2 items

3-4 formatting mistakes 5+ formatting errors present

/4

Content and Development

The essay adequately contends with the subject matter and offers a clear and concise discussion the prompt. Clear thesis is present.

The essay deals with the subject matter, but fails to address entire prompt. Thesis is present.

The essay does not adequately address the subject matter or the prompt. Thesis is weak or not present

The essay shows a lack of understanding of the subject matter or depth of thought. Attempt at a thesis is completely absent.

/4

Organization

Essay flows and transitions well; Logical paragraphing; Introduction and conclusion present

Essay flows generally well, with only a few choppy areas

Ideas in the essay do not seem to be linked very well. Essay is difficult to understand.

Essay construction impedes the purpose and meaning of the essay

/4

Total /32 *100=

______%

Morphing Your Rubric(s)Be sure your items are aligned with

objectives and department normsReview your rubric for clear language

and explicit (understandable) description of proficiency levels

Take one set of assignments and array them on Excel to find inconsistent items

Any inconsistent items should be expanded so you can pinpoint what is not working for students

Morph Your InstructionIf you notice there is something on

the rubric that students are consistently unsuccessful at, then you can spend extra time on that in class instead of on items they have already mastered.

The morphed rubric will help you measure whether your instruction is helping or if you need to choose another tactic.

Is it Working?Student scores on each item should be

consistent with their total scores.If not, keep morphing the rubric/instruction.Look for growth in scores over the

semester, if you are using the same rubric each time. (This also helps students understand consistent expectations)

Keep in mind student lives will occasionally impact their scores over the semester, so don’t freak out if all students don’t show improvement all the time.

Final ThoughtsQuestions?This presentation is linked at

ajahenriquez.wordpress.comA screencast of how to manipulate data in

excel are available at ◦ http://www.youtube.com/watch?

v=nDez8VmlN9A

ReferencesBizzell, P. (1982). Review: College

Composition: Initiation into the Academic Discourse Community. Curriculum Inquiry, 12(2), 191-207.

rubric. (n.d.). Collins English Dictionary - Complete & Unabridged 10th Edition. Retrieved October 20, 2011, from Dictionary.com website: http://dictionary.reference.com/browse/rubric