Embed Size (px)
Citation preview
Methodology and Data Qualityof the
ANES 2008-2009 Panel Study:
Lessons for Future Internet PanelsMatthew DeBell
Stanford University
2
Acknowledgments
• National Science Foundation• Stanford University & the University of
Michigan • Knowledge Networks collected data• ANES people:
– Jon A. Krosnick & Arthur Lupia, PIs (2005-2009)
– Vincent Hutchings, Associate PI (2005-2009)– Matthew DeBell, Stanford project director– lots of others!
This talk
• Describe an Internet panel study: ANES 2008-2009 Panel Study
• Evaluate data quality and related factors
• Methodological lessons
3 problems to frame the talk
3 problems to frame the talk
• Coverage error– Inaccurate representation
3 problems to frame the talk
• Coverage error– Inaccurate representation
• Nonresponse bias– Inaccurate representation
3 problems to frame the talk
• Coverage error– Inaccurate representation
• Nonresponse bias– Inaccurate representation
• Attrition– Loss of representation– Loss of power
Other issues
• Conditioning– Loss of representation
• Measurement error– Reporting error, satisficing, other
mode-related effects
This talk
• Describe an Internet panel study: ANES 2008-2009 Panel Study
• Evaluate data quality and related factors
• Methodological lessons
American National Election Studies
• Presidential elections in the USA• 1948-present• Standard mode is face-to-face• Two-wave panel: before and after
each presidential election
ANES 2008-2009 Panel Study
12
Design
• Sequential mixed-mode– Phone recruitment– Internet panel
• 21 waves online (25-30 minutes each)
– 7 “ANES waves” – 14 “off-waves” of non-political KN data
• Internet provided to Rs without access
13
MSN TV 2
14
Recruitment
• RDD recruitment– Landline only
• Two cohorts– Cohort 1 started January 2008– Cohort 2 started September 2008
• One person per household
• $10 per month incentive
• Recruitment-stage response rate (AAPOR RR3): 42%
15
Mixed-Mode Recruitment
• Internet-only recruitment fallback– Cases not enrolled by telephone (2,992)
were mailed a letter asking them to complete the recruitment survey on-line. 119 did so.
Communications (first 8 months)
• D-0 invitation email• D+3 email reminder• D+6 email reminder #2• D+14 phone reminder (repeated
weekly)
Revised communications
• D-3 notice email• D-0 invitation email• D+3 email reminder• D+6email reminder #2• D+11 email reminder #3• D+13 phone reminder (repeated weekly)
• D+19 email reminder #4 (repeated 10d)
This talk
• Describe an Internet panel study: ANES 2008-2009 Panel Study
• Evaluate data quality and related factors
• Methodological lessons
19
Part 2: Quality Evaluation
• Will show:– Number of cases– Dropout recovery– Case validation – Response rate– Panel retention / attrition numbers– Attrition effects– Accuracy of estimates
20
ncohort 1 cohort2
• Wave 1 (Jan ’08) 1,624• Wave 2 (Feb ’08) 1,458• Wave 6 (Jun ’08) 1,421 • Wave 9 (Sep ’08) 1,488 + 1,106 = 2,594• Wave 10 (Oct ’08) 1,511 + 1,126 = 2,637• Wave 11 (Nov ’08) 1,508 + 1,167 = 2,675 • Wave 13 (Jan ’09) 1,453 + 1,094 = 2,547• Wave 17 (May ’09) 1,387 + 1,016 = 2,403
21
Panelist Recovery Experiment
Table 1. Yield and retention percentages in Panel Study incentive groups, by month
Month Total $30 group $50 group Difference Total $30 group $50 group DifferenceJune 48.5 45.0 52.0 7.0 — — — —July — — — — — — — —August 55.5 49.0 62.0 13.0 † 77.3 68.9 84.6 15.7 *September 13.5 11.0 16.0 5.0 21.6 17.8 25.0 7.2Notes:— not applicable.† statistically significant at p < .10.* statistically significant at p<.05.Initial N=100 in the $30 group, 100 in the $50 group, and 200 overall.The September figures reflect only the first two days of data collection.Yield is the proportion of the invited panelists who completed the surveyRetention is the proportion of invited panelists who completed the June survey and completed a later survey
Yield Retention
22
Panelist Recovery Action
• $50 offers to all 282 panel “dropouts”.– Dropouts had completed the Profile or
Wave 1 or Wave 2 but not waves 7-9.
• Result highlights:– W10: 132 completions (47 percent)– W11: 132 completions (47 percent)– W17: 129 completions (46 percent)
23
Case Validation
• Telephone surveys with 10% subsample of Rs to each of the 7 planned ANES waves
• 1,482 interviews
• We found…
24
Case Validation
• Telephone surveys with 10% subsample of Rs to each of the 7 planned ANES waves
• 1,482 interviews
• 100% confirmed names and participation
25
But…
• Only half to two-thirds recalled topic of previous survey– Probably due to excessive delay in
validation calls.
• Imperfect item reliability– 18 instances of sex inconsistency– 91 instances of year of birth inconsistency
• Many appear to be data entry errors
• Error rates appear consistent with face-to-face surveys
26
Response Rates
RR1 RR3 RR5min est max
• Recruitment 26 42 75• Wave 1 18 29 51• Wave 2 17 26 46• Wave 6 16 25 45 • Wave 11 17 27 47• Wave 17 15 24 43
Retention: ANES vs. KnowledgePanel
• ANES panel retention was better than KP– [graphic & numbers not for public
distribution]
27
28
Retention Rates at Wave 11
• n = 2,675
• Retention– From Recruitment 63 percent (2,649)– From Profile 84 percent (2,439)– From Wave 1 85 percent (1,381)– From Wave 10 95 percent (2,500)
29
Completions at Wave 11 (Nov ’08)
total cohort 1
• Total 2675 1508
• Completed: – Recruitment 2649 (99%) 1483 (98%)– Profile 2439 (91%) 1328 (88%)– Wave 1 1381 (52%) 1381 (92%)– Wave 10 2500 (93%) 1433 (95%)– Waves 9 & 10 2319 (87%) 1345 (89%)– All ANES stages 1058 (40%) 1058 (70%)– All stages (1-10) 738 (28%) 738 (49%)
30
Other Retention Rates
• Mean wave-to-wave retention: 91 percent
• Cohort 1 Rs who completed all ANES waves through May 2009 (1,2,6,9,10,11,17): 68 percent (939)
• 83% of May 2009 completers had completed Waves 9, 10, and 11.
31
Attrition effects
• 1623 Rs completed Wave 1• 1,258 of these also completed
Wave 17– 78 percent retention; 22 percent
attrition
• Ran frequencies on all Wave 1 variables for these two groups and…
32
Attrition effects
• 1623 Rs completed Wave 1• 1,258 of these also completed
Wave 17– 78 percent retention; 22 percent
attrition
• Ran frequencies on all Wave 1 variables for these two groups
• Average difference: 1.3 points.
Differential attrition (wave17)
Factor W1 W17 Diff.•18-29 year olds 19.716.1 -3.6•Males 47.244.4 -2.8•HS dropouts 10.88.8 -2.0•Home renters 17.414.7 -2.7•Non-voters 22.718.2 -4.5•No Obama affect 30.527.4 -3.1
34
Accuracy of Estimates (1 of
2)• Benchmark to CPS.• 43 statistics examined, for:
– Age, sex, race, ethnicity, race/ethnicity, education, home tenure, household size, marital status, household income, presidential vote choice, voter turnout
• Estimates are within 5 points of benchmark for 84 percent (36 of 43) of statistics examined.
Accuracy of Estimates (2 of
2)• >5pt errors for a few statistics
– Renters (-9.3)– One-person households (-5.4)– Married (+10.2)– Income >$100,000/yr (-6.6)
• Average error: 2.1 points
This talk
• Describe an Internet panel study: ANES 2008-2009 Panel Study
• Evaluate data quality and related factors
• Methodological lessons
Part 3: Lessons
Concerning:•Data quality•Measuring quality immediately•Promoting quality
Data Quality
• Data quality from the telephone-recruited panel is consistent with expectations for a high-quality telephone survey– Good accuracy– Only moderate attrition
Measuring Quality Immediately
• Monitor attrition during panel• Validation interviews
– Should happen immediately after web completion
• Concern: respondent identity
Promoting Quality
• Fight non-response bias – high-quality recruitment– multiple contacts over a long period of time– incentives
• Fight attrition– pleasant content– incentives– dropout recovery with added incentives
• Fight conditioning with variety
Final Thoughts
• Other possibilities– Targeted incentives to combat NR
bias– Oversamples at recruitment to
combat NR
• Landline RDD is obsolete!• Literacy
