2011 STAR Nuts and Bolts Riverside County Assessment Network March 4, 2011

1

2011 STAR Nuts and BoltsRiverside County Assessment Network

March 4, 2011Wes Scott

Key Data Systems

2

Overview

• What is new in 2011• Who Takes What Test• To Be Gridded by All Students• Hand-gridding demographic data• Irregularities and Post test gridding• Scorables and non-scorables packaging• Special Education issues• Considerations and Reminders

3

What is new in 2011

• CMA is now... – ELA: Grades 3 – 11– Math: Grades 2-7, Algebra I, and Geometry– Science: Grades 5,8 and 10

• Grade 4 Writing is back• 2 minor editorial changes in box A3 of the

answer documents

4

CMA• ELA grades 3-11, Math grade 3-7, Algebra I, Geometry, Science 5, 8 & 10• May test CMA in one content area and CST in another• Can not test CMA and CST in same content area• If CMA, can not assess with CAPA• Take grade 4 or 7 CMA writing if taking CMA ELA• One more test section than CST, all content areas, all grades• Fewer answer options (3 in CMA versus 4 in CST)• Fewer items

– Grade 3-5 – 57 items, 48 count– Grade 6-8 – 63 items, 54 count– Grade 9-11 – 70 items (Grade 10 science is 66 items), 60 count

• Accountability– CMA grades 3-8, ELA will count in AYP– CMA grades 3-9 in 2010 Base API & 2011 Growth API– CMA Geometry, grade 8, will count in AYP, December release, not in API

5

CMA (cont.)

• Same participation criteria as last year– CST scores in any previous year of Far Below Basic

or Below Basic OR Scoring Proficient or Advanced on TWO previous CAPA administrations

– Multiple measures of assessments– Determined will not attain grade level proficiency– Response to grade level instruction– IEP must state the student will take the CMA, and

in which content area, before STAR testing

6

Answer Document Change

• Box A3 added subsection identification (this is a grade 10 answer document)

7

Answer Document Review• Box A2

– There is no ‘All Tests’ column

– In alphabetical order– A column for each

possible test for this grade level (varies with grade level). This sample is grade 5.

8

Answer Document Review (cont.)

• Box 7c on CST answer document. Used to identify which CMA tests will be taken and scored

• Separate answer document for CMA Grades 3, 8-11• Grades 4 -7 answer documents include CST and CMA

answer sections...CST on left and CMA on right.• CST and CMA version numbers at top right portion of

page. Required – sets scoring key.• CMA section has box around it and different icons.

9

CMA Version

Box around CMA answer section

Grade 5 Answer doc

10

Race/Ethnicity• Box 8 – Hispanic/Latino – Yes/No bubbles• Box 9 – Race. To be filled in regardless of

box 8 answer (Federal rules)• California says “do not guess” if race is blank• API, AYP, If Box 8 is ‘Yes’, will be Hispanic

regardless of Race bubbles• If Box 8 is ‘No’ and no Race, will not be in

any ethnic group.• If Box 8 is ‘No’ and bubbles in two or more

racial groups, ‘Two or More Races’ subgroup• Note: races listed alphabetical and not by

group

11

Who Takes What Test• All students take CST, CMA or CAPA, except…

– If student already tested in a previous school– Student enrolled after the first day of testing (testing optional)– Parent Exemption

• If less than 85% tested in any grade level test, then the school will not get an API. Grade level tests include:– ELA grades 2-11– Math grades 2-9– Science in grades 5, 8 and 10– HSS in grades 8 and 11

• AYP– Grades 2-8 require 95% participation– High school without a grade 10, STAR results are used

12

Who Takes What Test (cont.)• Math grades 7-11 (pages 15-17, D-1 in District/Site

Coordinators Manual)– Take End-Of-Course (EOC) test if in standards aligned course

• Grade 7 – Grade 7 Math or Algebra I• Grade 8 & 9 take EOC Math or General Math (based on what Math course

the student will complete this school year)• Grade 10 & 11 take EOC Math unless not in a standards aligned course

(e.g. Business math)• If passed Algebra I and II and Geometry (new in 2010) or Integrated Math

III and not taking any Math course OR taking higher-level math, take Summative Math

• Grade 10 & 11, take no Math test if will not complete standards aligned course and don’t qualify to take the Summative Math test

• If no EOC Math test taken, API score of ‘200’ (Far Below Basic) at a reduced weight (.10 versus .20)

13

Who Takes What Test (cont.)

• Science grades 9-11 (pages 17-18, D-2)– Grade 10, all students, take Life Science test– Take EOC test if in standards aligned course

• Some district flexibility on this.– AP courses take corresponding CST– Higher level science, may take most closely aligned CST

• No EOC Science test– API score of ‘200’ (Far Below Basic) at a reduced rate (.05

versus .22)– If take Grade 10 Life Science test, no ‘200’

– Some grade 10 students take two Science tests

14


• HSS Grades 9-11– World History is an EOC test.

• Assess when student takes World History course• May take more than once if retaking course• If taking in grade 11, will take two HSS tests

– Grade 11 – U.S. History (all grade 11 students)

15


• Can retake EOC Math and/or Science if…– Student failed course and is retaking the course– Student is taking AP course after completing the

standards based course (e.g. AP Chemistry)– Student is taking General Math in grades 8 and 9– Student is taking the Summative Math test

16

STS• Grades 2-11, ELA and Math• Taken in STAR testing window• STS Math, grades 2-7, Algebra I and Geometry• Take Algebra I/Geometry if taking equivalent CST

math test• Same criteria for who takes STS

– English Learners in US school less than 12 months (cumulative)

– EL’s receiving academic instruction in Spanish– Any Spanish speaking EL may take STS (district decision)– Non-EL’s (English Only, IFEP, RFEP) can NOT take STS

17

To Be Gridded by All Students• Box 1 – student name, teacher, etc.• Box 7 (grade)• Box 7c – CMA only• Grade 7, page 2, Math test taken, version (Algebra 1).

Will be scored as 7th grade math if not bubbledNote Special stopping place depending on Math test taken for CMA

• Grade 4-7, page 3 – CMA version (if applicable)

18

To Be Gridded by All Students

• Grades 9-11– Page 2 – CST Science test taken and version– Page 3 – CST Math test taken and version (grades

8-11)– Page 2 – HSS version

• Grade 11, page 2, EAP-ELA (bubble releasing EAP scores)

• Grade 11, page 3, EAP-Math (bubble releasing EAP scores)

19

Box 1

Hispanic or Latino

Grade

CMA – 7c

SSID

20

EAP (CSU and CCC)

World History

Version

ELA Version

21

EAP

Science Taken and Version

Math Taken

And Version

22

Hand-Gridding Demographic Data• Why hand-grid an answer doc?

– Student is new to the district since pre-id• Alternative - Late pre-id up to one week before end of testing

window– Student changes schools or tracks after pre-id submitted

• STAR web site option to change schools – only need to change CBEDS

– Will result in a late label being created by ETS.• Can take pre-id document to new school – online update CBEDS• If change to new wave, void first pre-id and do late pre-id in new

wave– Late pre-id available until week before end of testing

period

23

Hand-Gridding Demographic Data (cont.)

– Believe pre-id data is incorrect• Alternative - Online data correction available until last day of

testing

– Pre-id answer document becomes unusable– Pre-id for incorrect assessment (e.g. should be CMA, not

CST)• If should take CMA and pre-id CST, or visa versa, and grade 3• If CMA but pre-id CST, grade 4-7, use pre-id answer document• If CMA but pre-id CST, grades 8-11. Hand-bubble CMA document

– Only bubble boxes 1-7c, 11 and 12– Be sure SSID and Name spelling exactly matches pre-id document

24


• Gridding by District staff – not students• All data is important – leave no blanks except those that don’t

apply to student (e.g. no RFEP info if not RFEP student)• Accuracy is vital as it impacts AYP and API• Box 10, Parent Education

– Similar Schools (Parent Ed has most impact of any component)– Socio-economically disadvantaged subgroup

• Box 11, student id, do NOT add zeros on the right side.• Box 12, SSID, always 10 digits, never starts with zero

– Used for matching Writing, CST, CMA, EAP• Grid all available bubbles (e.g. 2 bubbles for day of month

and all bubbles for years), ETS does not default any values

25


• Box 14 (CBEDS). If left blank, CDE treats as YES – may be bad• Box 16c (Special Ed Exit Date). Only if 16a is ‘000’. Very important for AYP• Box 20 – only for students who live outside district but attend district to

receive special ed services per an IEP. Must be valid county/district code• Box 22 (RFEP Proficient/Advanced 3 times on CST ELA, SINCE RFEP date).

If left blank (grades 5-11), CDE treats as YES – bad. Has to be blank for grades 2-4.

• Boxes 15 (English Proficiency), 22 (RFEP CST ELA 3 times) and 24 (Date first entered a U.S. School) used to determine EL subgroup

• 3% rule: Gender, Hispanic or Latino, Parent Ed, SSID, CBEDS, English Proficiency, Disability code, NSLP (boxes 6,8,10,12,14,15,16a,23)– If more then 3% blank for a school, ETS scoring is stopped– District must correct (cost is $1.32 per student corrected)– Corrections need to be made within 7 days (approximately)– Only after corrections made will processing continue.

26

CBEDS

English Proficiency

Special Ed Info

RFEP 3 Year

EL Date First Enrolled in USCounty/District

of Residence

Special Ed Exit Date

RFEP Date

NSLP

27

Most Frequent Errors• Missing CBEDS (D-283/S-265)• Missing or bad EL Date (487)• Missing RFEP 3-Year Prof/Adv (141)• Missing NSLP (924)• Missing Parent Education (751)• Missing English Proficiency (532)• Missing Hispanic indicator (235)• Missing Special Ed Exit Date ???• Bad IEP County/District of Residence (262,

145=current district)

28


• Common questions– What is continuous enrollment?

• Enrolled on or before CBEDS Information day (first Wednesday in October) and has not been disenrolled since that date

• Requires rigorous record keeping• Used to determine who is in API and Proficiency on AYP

– What value should be used for ‘Date First Enrolled in a US school’ if they were born in the U.S.?

• When they first entered school…likely kindergarten

29

Irregularities• See Appendix C (page 76) for guidelines and reporting forms• Irregularity – helping students or security breaches

– Examiner reading the test booklet (except grade 2)– Anything left on the classroom walls that will help a student– Test booklets not being locked up– Test booklets can NOT be taken out of the room except when the

teacher is returning them to the secure site– Preview/review of test booklets by anyone (exception for grade 2

teachers on the first day of testing)– Teacher interpreting items for students– Test booklets being used by more than one student

30

Irregularities (cont.)

• Report irregularities and Inappropriate Test Preparation to the Site Coordinator who reports it to the District coordinator

• District coordinator fills out the form (no student or teacher names are to be mentioned) and emails or faxes to the CDE (no calling)

• CDE will tell District coordinator which bubble, if any, to fill in box A1

31

Post Test Gridding

• Be sure Box 1 is complete• Be sure all student gridding is complete and

accurate (see earlier slides)• If hand-gridded demographics, be sure it is complete (AYP and API impact and/or big dollar cost to correct later)• Boxes, A2, A3 and A4 – See last page of DFA for listing of codes (content varies with grade

and test)

32

Post Test Gridding (Cont.)

• Box A2 – Testing Conditions– Valid values

• Pages 115-116

– Significantly impact AYP & API• E,L,M,T exclude from AYP

enrollment, unless...

– Sample is grade 5

33

A2 Accountability Impact AYP API

Enroll Tested Proficient Valid Score

A-Absent N N*

C-Cheat N B*

E-Medical Emergency N N*

F-Sick – did not complete ? N* ?

L-Enroll after first day N^ N*^

M-Enroll first day then moved N^ N*^

P-Parent Exempt N N*

T-Previously Tested N N

X-Refused acc/mod n/a n/a n/a n/a n/a

Y-Replaced n/a n/a n/a n/a n/a

Z-Not attempt N

? = Depends on how many items attempted and score

B* = Include in Base API, exclude from Growth API

N* = Excluded if no items attempted

N^ = Excluded unless CBEDS=Y

34

A2 – Errors

• L – Enrolled after first day and tested– ELA – 71, 13 had CBEDS=Y– Math – 62, 13 had CBEDS=Y

• T – Tested in a previous district– ELA – 26, 4 had CBEDS=Y– Math – 20, 4 had CBEDS=Y

• Most of L and T above were not proficient

35

A2 – Errors (cont.)

• A – Absent, 192-ELA scores, 109-Math scores• E – Medical Emergency, 5-ELA, 4-Math scores• P – Parent Exempt, 41-ELA, 3-Math scores• M – Enrolled first day of testing, moved before

test was given, 38-ELA, 58-Math scores• A, P, and M will be included in AYP/API.

– E is excluded from AYP regardless of scores

36

Post Test Gridding (Cont.)• Box A3 – Accommodations/Modifications (page 83-87 &

143 in District Coordinators Manual)– If in IEP and used in instruction, student must be allowed

to use accommodation/modification– If not in IEP, can NOT use it– ONLY mark if student was observed using it, if not used,

mark box A2, row X.– Accommodations do not impact AYP or API– Modifications count as Far Below Basic for API and ‘not

tested’ for AYP• A4 – English Learner Test Variations

– No IEP required– English Learners Only– Glossary/Word list

• Can not use for ELA• Translations only – no definitions

37

Scorables and Non-scorables Packaging

• Scorables (orange label)– All answer docs (includes grades 2 & 3 test booklets)– *If Grade 3 absent, pre-id CMA & CST, only return one

answer document with the scorables.– SGID sheets (more on this later)– Master File Sheets– Processing Form (district only)– See pages 40-42 of the District and Test Site Coordinators

Manual for a comprehensive list and the required packing order

38

Scorables and Non-scorables Packaging (cont.)

• Non-Scorables (green label)– CAPA Examiner’s Manuals– Grade 2 DFAs– Test booklets– Voided answer docs– Braille and Large print booklets

39


• Following packaging instructions to the ‘T’– 1 Master File for school for each admin– SGID for each grade (one per teacher if want

teacher reports) and one for CAPA– CAPA on top of lower grade (do not return

stimulus cards)– Lowest grades on top– Highest grades on bottom– STS under CST

40


• SGID– Sets school code and grade for all answer documents (that’s why one

can move pre-id answer docs with student if they change schools)– *Filling in teacher name on back will save that data in the results file– School code must be ‘0000001’ for NPS– No grade for CAPA– Back

• Instructional Materials for grades 2-8 only– CAUTION: If SGID grade and answer doc grades are different

• Student counts as participating• Student will receive no scores

41


• When return Multiple-Choice and CAPA– Site: 2 working days after last test for that

administration– District: 5 working days after last test for that

administration– Keep administrations separate– Track all documents, from original receipt until

shipping back to ETS, in order to not have to pay excessive order charges

42

Special Education Issues

• No Below Level Testing• Be sure all Special Education students are

tested• CAPA and CMA

– Scores count the same as CST for AYP and API– Assess in STAR testing window– Assess all who qualify– Appropriate test for the student– Can only help AYP and API

43

Special Education Issues (cont.)

• CAPA and CMA limits – applied in March 2011 AYP release– CAPA 1% rule (AYP only)

• Applies to non-mobile students only• If number Proficient or Advanced is more than 1% tested in district,

those over 1% are considered not proficient for AYP purposes• *File exception request (due around July 1) if even close to 1%

– CMA 2% rule (AYP only)• Applies to non-mobile students only• If CMA Proficient or Advanced is more then 2% tested, those over 2%

are considered not proficient for AYP purposes• No exceptions allowed

– CAPA and CMA 3% rule (AYP only)• Combined CAPA and CMA can not exceed 3%• If CAPA exception granted, ‘combined’ limit is 2% plus CAPA exception

amount.

44


• CMA– Grade 4-7 answer document, cover up or fold

under CST answer page when testing CMA– Test separately from CST because CMA DFA and

number of sections is different than CST– Coordinate schedule with CST since some students

take CMA and CST (e.g. Grade 8 CMA students will take some CSTs (at least HSS))

45


• Accommodations and Modifications– See Pages 83-87,143 for complete list

• Separate list for CST (page 83-87) and CMA (page 143)

– IEP can not exempt a student from testing– Accommodations

• Result in a valid test• Must match instructional accommodations

– Modifications change test interpretation• Actual score on individual student reports• Far Below Basic for API• Considered not tested for AYP

46


• CMA Accommodations that are CST Modifications– ELA

• American Sign Language/Manually Coded English to present questions

• ELA Test questions/answer options read aloud

– Math• Use of Calculator on grade 5 math test• Math Manipulatives on Math or Science tests

• If modification required, must take CST (e.g. use dictionary)

47

Testing Considerations• Test early in the day• Test with normal teacher if possible (especially for Elementary)• Consider not testing on Monday and Friday• Allow plenty of time for make-ups• Tests are untimed (within reason) – set up room for those needing

additional time• Grades 2 and 3, limit to one section per day• No breaks within a section except supervised bathroom breaks• Multiple sections must be administered sequentially• Multiple sections in a day require a 10 minute break between sections• If complete a section, after starting the next section, can not go back and

review answers in first section• Do not administer CMA and CST in the same room at the same time

48

Additional Reminders• Distribute non-secure DFAs (grades 3-11) before first day of testing• Grade 2, secure version, DFA (includes items) can NOT be distributed

before first day. Non-secure version is available on STAR web site.• *Grade 2, all test booklets need to be the same version in a classroom• Distribute test materials first day of testing…not earlier• Have student write their name on the answer document and all test

booklets• Be sure students only get their answer doc and test booklets• Students can write in test booklets

– Grade 2 & 3. Erase extraneous marks. Recommend no use of Highlighters. If too much bleed-through, transcribe to a new answer booklet and ‘void’ the original answer booklet

• Excessive erasures (all grade levels) raises a red flag for CDE• Immediately report possible irregularities• Administrative errors (page 78 & Form A-5 page 80) need to be

documented but are not reported to the CDE

49

Additional Reminders (cont.)• Box A3. Note that each DFA lists only accommodations/modifications

applicable to that grade level/test. Writing has additional codes.• *Unlisted Accommodations/Modifications must be cleared through the

CDE prior to testing…do it 30 days before• Grade 2, teacher read items…do not mark as ‘read aloud’ modification

when DFA says to read the item aloud.• Directions for Administration (DFA)

– Must read ‘Say’ boxes exactly as written– ‘T’ means can translate ‘Say’ box

• *No modifications allowed for CMA. If modification required, must take CST

• Before testing, give teachers list of any demographic information needing to be gridded.

• Request ETS to send data as ASCII text, not XML• There is a late processing fee of $385 if ‘extra’ answer docs sent in late and

ETS has already processed the district data

50

Additional Reminders (cont.)

• Miscellaneous– CMA – Braille for grades 3-8 except for Algebra I– Call ETS if running short on materials. They can

deliver quickly using Supplemental Order process (need to call ETS)

– Be aware of excessive order charge (if use less than 90% of materials ordered...ETS overage is excluded from calculation).

– No Teacher Reports– API includes CAPA science (grades 5,8 and 10)

51

Final Thoughts

• Be familiar with the following. They are all available on the STAR web site.– Directions for Administration (all grades and test

types)– District and Test Site Coordinator Manual– Sample answer documents

• Maintain strict security• Accuracy of demographic and other gridded

data on the answer document is vital

52

Contractor Contact Information

• STAR (ETS)– TAC: 1-800-955-2594– FAX: 1-800-541-8455 or 1-925-808-2118– [email protected]– www.startest.org

• CDE STAR Office– Phone: 1-916-445-8765– [email protected]– www.cde.ca.gov

53

QUESTIONS?

[email protected] x 212

Documents

2011 STAR Nuts and Bolts Riverside County Assessment Network March 4, 2011