Embed Size (px)
Citation preview
September 23-26, 2014 Ramada Hotel Berlin-Alexanderplatz
Organized by
#MRMW
Title Sponsor Gold Sponsors
Silver Sponsors
Premier Sponsor Bag Sponsor Workshop Sponsors
MRMW Berlin Sept. 23-26, 2014 #MRMW
Association & Media Partners
Event App Partner
Networking Reception Partner Exhibitor
MRMW Berlin Sept. 23-26, 2014 #MRMW
Exploring optimal survey design for mobile web – a scientific
perspective
1
perspective
September 23-26, 2014Berlin
Aigul Mavletova
NRU Higher School of Economics, Russia
OMI – Online Market Intelligence
In collaboration with:
2
Mick Couper, Institute for Social Research,University of Michigan
Outline
The main focus: breakoff rates
•Scrolling vs. paging design
•Scrolling design:o the number of questions presented on the pageo survey with user-controlled skips vs. survey without
3
o survey with user-controlled skips vs. survey withoutskips in the scrolling design
•SMS vs. e-mail
•Meta-analysis of breakoff rates
Breakoff rates: non-optimized vs. PC vs. mobile optimized
5%
10%
15%
20%
25%
30%
35%
40%
Diff=2
Diff=2.1
Diff=2.2Diff=3.5
5%
10%
15%
20%
25%
30%
35%Diff=1.1
Diff=1.9
Diff=4.7
Diff=9
Non-optimized vs. PC Mobile optimized vs. PC
4
0%
5%
Bosnjak et al., 2013
Guidry, 2012 McClain et al., 2012
Schmidt and Wenzel, 2013
0%
2%
4%
6%
8%
10%
12%
14%
16%
18%
20%
Baker-Prewitt, 2013 Peterson et al., 2013 Stapleton, 2013 Wells, Bailey, and Link, 2013
Non-optimized mobile
PC
Mobile optimized
Diff=3.6
Diff=2.6 Diff=3.3
Diff=1.3
Diff=5.3
Diff=2.7
Diff=5Diff=4
0%
5%
Buskirk and Andrus, 2014
Lattery, Park Bartolone, and Saunders, 2013
Mavletova and Couper, 2013
Mavletova, 2013
Participation rates and completion times
Participation rates twice lower in mobile than PC web
Completion times ≈ 50% longer in mobile than PC web
5
43%
79% 84%
50%
60%
70%
80%
90%
Mobile traffic and breakoff rates
6
23%
43%
18% 17%
0%
10%
20%
30%
40%
2012 2013 Q3’12 Q3’13
Mobile traffic Breakoff rates
MobilePC
•20% mobile respondents
Mobile breakoff rates: 41%PC breakoff rates: 24%
Should we care?
7
≈ 3 million partial mobile web interviews per year (Jue andLuck, 2014)
≈ 10 million annual breakoffs in the US
Survey design: paging vs. scrolling
The overall layout should minimize the need for scrolling (eitherhorizontally or vertically) to the extent possible.
Report of the AAPOR Task Force on EmergingTechnologies in Public Opinion Research (April, 2014)
8
horizontally or vertically) to the extent possible.
The number of questions per screen should generally be two orless to minimize scrolling.
Paging vs. scrolling design
McGeeney & Marlar (2013): higher breakoff rates in thepaging version than in the scrolling version (13% ofmobile respondents; a 7-item and 13-item survey)
9
mobile respondents; a 7-item and 13-item survey)
Paging Scrolling
Experimental design
17 pages 2 pages
•Volunteer online access panel - Online Market Intelligence
10
•Volunteer online access panel - Online Market Intelligence
•April, 2013, Russia
•Software: Unipark
•2,110 respondents: 4,000 invitations, participation rate=52.8%
(1 invitation, 1 reminder)
Experimental design
Paging Scrolling
11
Participation and breakoff rates
53.8% 49.9%
8.0%+
51.8% 48.6%
10.2%+20.0%
30.0%
40.0%
50.0%
60.0%
ScrollingPaging
12
74% - smartphones9% - feature phones3% - tablets14% - PC
8.0%+10.2%+
0.0%
10.0%
Participation rate Participation rate (mobile devices only)
Breakoff rate
Chi-square (df=1), + p<0.07
Scrolling vs. paging
4.5*** 4.9***
9.1***6.6***
0123456789
10
Objective mean completion time Subjective evaluation of
ScrollingPaging
Completion times (minutes)
13***p<0.001
Objective mean completion time Subjective evaluation of completion time
18.9%***
1.8%
25.8%***
1.4%0.0%
5.0%
10.0%
15.0%
20.0%
25.0%
30.0%
Technical difficulties Item nonresponse
ScrollingPaging
Technical difficulties and item nonresponse rate
Implications
Scrolling design makes the process of completing the surveyeasier in mobile web mode.
14
Scrolling - 2
ü Does the number of questions presented on the page have an effect on data quality in a scrolling design?
Research questions
15
ü Is it different in a survey with user-controlled skips and in a survey without skips in the scrolling design?
Skips
Experimental design
Without skips
With skips
16
GO TO Q8
Skips
Experimental design
Number of itemson page
5 items (6 pages)
15 items (2 pages)
30 items (1 page)
Without skips
With skips
17
ü Volunteer online access panel - Online Market Intelligence
ü 10 minute survey
ü October, 2013, Russia
ü Software: Unipark
ü 7,740 SMS invitations, 2,032 respondents, participation rate=26.8%
Participation and breakoff rates
26.1%
13.1%
26.4%
13.4%
10.0%
15.0%
20.0%
25.0%
30.0%
Without skipsWith skips
18
0.0%
5.0%
Participation rate Breakoff rate
91% - smartphones5% - feature phones3% - tablets2% - PC
Breakoff rates
10.7%13.3%
15.3%13.5% 12.7%
14.0%
8.0%
10.0%
12.0%
14.0%
16.0%
18.0%
Without skips With skips
19
0.0%
2.0%
4.0%
6.0%
30 items per page
15 items per page
5 items per page
30 items per page
15 items per page
5 items per page
7.7 8.5 8.76.6 7.3 8.0
0
2
4
6
8
10
30 items per page
15 items per page
5 items per page
30 items per page
15 items per page
5 items per page
With skips***Without skips**
Completion times (minutes)
20** p<0.01, *** p<0.001
7,3%
4,7%3,3%
7,8% 7,2%
1,9%
0123456789
30 items per page
15 items per page
5 items per page
30 items per page
15 items per page
5 items per page
Item nonresponse rate
Implications
üSurvey with skips do not produce higher breakoff rates.
üThe optimal trade-off between breakoff rates, completion
times, and item nonresponse rates:
21
15 items per page in the survey without skips
5 items per page in the survey with skips
Experimental design
Invitation Reminder
SMS E-mail E-mailwithout URL
SMS vs. e-mail
22
SMS + + +E-mail + + -
Questionnaire:17 items
Participation rates
52.7% 51.3%***43.7%***
30.0%
40.0%
50.0%
60.0%
SMS
23 ***p<0.001
10.5%***5.7%***
0.0%
10.0%
20.0%
Participation rate Participation rate (mobile devices only)
Breakoff rate
Participation rates
60.9% 54.6% 53.2% 48.7%
44.1%
53.5% 52.4% 52.1%47.7%
32.8%30.0%
40.0%
50.0%
60.0%
70.0%
Participation rate
Participation
24p<0.001
0.0%
10.0%
20.0%
30.0%
E-mail-SMS SMS-E-mail SMS-SMS SMS-E-mail without URL
E-mail-E-mail
Participation rate (mobile devices only)
Implications
üSMS invitation is more efficient for increasing participation
rates in mobile web.
üUsing e-mail invitation-SMS reminder modes increases the
25
overall participation rates.
Meta-analysis: breakoff rates
Inclusion criteria:
(1) Based on online panels (probability and non-probability)
(2) Assigned to mobile web survey mode without having anopportunity to choose the device or could select their preferreddevice
26
(3) Studies could be either browser-based or app-based
(4) Available relevant statistics on breakoff rates and moderators
→ 14 studies with 39 independent samples
Forest plot: breakoff rates (log transformed proportions)
27
Significant factors (odds ratios)
0.70.5
0.70.9
0.40.6
1.21.4 1.3
1.9
0.2
0.4
0.6
0.8
1
1.2
1.4
1.6
1.8
2
28
0
0.2
Implications
Breakoffs can be decreased by:
üMobile optimization
üLess complex survey design
üShorter surveys (modular mobile web surveys?)
29
üLarger number of reminders
üScrolling design
Participation rates can be increased:
by SMS invitation (or by sending e-mail invitation-SMSreminder)
Title Sponsor Gold Sponsors
Silver Sponsors
Premier Sponsor Bag Sponsor Workshop Sponsors
MRMW Berlin Sept. 23-26, 2014 #MRMW
Association & Media Partners
Event App Partner
Networking Reception Partner Exhibitor
MRMW Berlin Sept. 23-26, 2014 #MRMW
September 23-26, 2014 Ramada Hotel Berlin-Alexanderplatz
Organized by
#MRMW
