Techniques and tools for field-based early-stage study and iteration of Ubicomp applications

Techniques and tools for field-based early-stage studyand iteration of Ubicomp applications

1

Scott CarterEECS DepartmentUniversity of California, Berkeleysacarter@cs.berkeley.edu

CommitteeJennifer Mankoff (chair), Anind Dey, John Canny, Peter Lyman

2

what are ubicomp apps? examples…social    proximity       jabberwocky (paulos ’04)

   location       urban tapestry (lane ‘05)

   activity

CarterTechniques and tools for field-based early-stage study and iteration of Ubicomp applications

resource queries    transit services   tour guide (abowd ’97)

3

capture and access    field data        health (alivetech ’05)    reminder apps

CarterTechniques and tools for field-based early-stage study and iteration of Ubicomp applications

gaming    cysmn (blast theory ’01)

   location,proximity-based

what are ubicomp apps? examples…

4

CarterTechniques and tools for field-based early-stage study and iteration of Ubicomp applications

ubicomp evaluation is hard (Carter ’05)

5

CarterTechniques and tools for field-based early-stage study and iteration of Ubicomp applications

ubicomp evaluation is hard

rapid prototyping

6

CarterTechniques and tools for field-based early-stage study and iteration of Ubicomp applications

ubicomp evaluation is hard

rapid prototyping

scale

7

CarterTechniques and tools for field-based early-stage study and iteration of Ubicomp applications

ubicomp evaluation is hard

rapid prototyping

scale

unobtrusiveness

8

CarterTechniques and tools for field-based early-stage study and iteration of Ubicomp applications

ubicomp evaluation is hard

rapid prototyping

scale

unobtrusiveness

data sparsity

9

CarterTechniques and tools for field-based early-stage study and iteration of Ubicomp applications

thesis

Leverage pre-existing infrastructures and the increasing power of mobile devices to address the challenges of early stage field evaluations of ubicomp applications.

10

CarterTechniques and tools for field-based early-stage study and iteration of Ubicomp applications

why early stage and field?

early stage

field

>> potential to iterate>> could answer some research questions

>> realism especially important in ubicomp >> Sharp & Rehman ‘05: “Real-world deployment is the only way”>> Kjeldskov: might be able to recreate field in a lab

11

CarterTechniques and tools for field-based early-stage study and iteration of Ubicomp applications

approach: WoZ

researcher designs app behavior in momento and provides instructions for wizards

wizards use momento to simulate app behavior

12

CarterTechniques and tools for field-based early-stage study and iteration of Ubicomp applications

approach: report methods

researcher designs experiment, configures momento or reporter

researcher runs experiment, monitoring tool

13

CarterTechniques and tools for field-based early-stage study and iteration of Ubicomp applications

limitations: when challenges do not applydata easy to get (no sparsity)

infrastructure already available (no scale issues)

target users early, early adopters (no unobtrusiveness)

tools off-the-shelf (no prototyping)

14

CarterTechniques and tools for field-based early-stage study and iteration of Ubicomp applications

most relevant related work

rapid prototyping

Topiary (Li ’04)

scale

unobtrusiveness

data sparsity

rapid prototypingContextESM (Intille ’03)

scale

unobtrusiveness

data sparsity

WoZ Report methods

SUEDE (Klemmer ‘00)

DART (Dow ‘05)

Doorman (Makela ‘01)

Crossweaver (Sinha ‘03)

Active Capture (Chang ‘05)

other work

iESP (Intel ’03)

Mobile probes (Hulkko, ‘04)

other work

15

CarterTechniques and tools for field-based early-stage study and iteration of Ubicomp applications

outline

ubicomp challenges case study

report methods

momento

validation plans

timeline

summary

16

CarterTechniques and tools for field-based early-stage study and iteration of Ubicomp applications

ubicomp challenges case study: dr. bluesocial mobile game

get points by collecting cures, healing the sick

17

CarterTechniques and tools for field-based early-stage study and iteration of Ubicomp applications

ubicomp challenges case study: dr. blue

18

CarterTechniques and tools for field-based early-stage study and iteration of Ubicomp applications

ubicomp challenges case study: dr. blue

19

CarterTechniques and tools for field-based early-stage study and iteration of Ubicomp applications

case study: dr. blue: rapid prototypingmajor difficulty: building a mobile application

communicationsensordisplay

also difficult: game engine

20

CarterTechniques and tools for field-based early-stage study and iteration of Ubicomp applications

case study: dr. blue: scale

everyone needs a device and recording method

21

CarterTechniques and tools for field-based early-stage study and iteration of Ubicomp applications

case study: dr. blue: unobtrusivenessnew device and interface

interruptions: evaluation and application behavior

22

CarterTechniques and tools for field-based early-stage study and iteration of Ubicomp applications

case study: dr. blue: sparsity

encounters may be limited

23

CarterTechniques and tools for field-based early-stage study and iteration of Ubicomp applications

outline

ubicomp challenges case study

report methods

momento

validation plans

timeline

summary

24

momento

WoZ simulation using SMS/MMS infrastructure

CarterTechniques and tools for field-based early-stage study and iteration of Ubicomp applications

25

momento

WoZ simulation using SMS/MMS infrastructure

… and J2ME on the mobile device

CarterTechniques and tools for field-based early-stage study and iteration of Ubicomp applications

26

momento

input and output

CarterTechniques and tools for field-based early-stage study and iteration of Ubicomp applications

27

momento: building dr. blue

CarterTechniques and tools for field-based early-stage study and iteration of Ubicomp applications

28

momento: building dr. blue

CarterTechniques and tools for field-based early-stage study and iteration of Ubicomp applications

29

momento: building dr. blue

CarterTechniques and tools for field-based early-stage study and iteration of Ubicomp applications

30

momento: building dr. blue

CarterTechniques and tools for field-based early-stage study and iteration of Ubicomp applications

31

momento: building dr. blue

CarterTechniques and tools for field-based early-stage study and iteration of Ubicomp applications

32

momento: limitations

SMS/MMS, wizard latency

CarterTechniques and tools for field-based early-stage study and iteration of Ubicomp applications

24-hour wizards?

synchronous and streaming data

gui creation (damask – Lin ’02)

momento: summary

CarterTechniques and tools for field-based early-stage study and iteration of Ubicomp applications

scale

unobtrusivenessdata sparsity

rapid prototypingany device supporting sms/mms

esm feedback on critical events

easy to change application behavior

integrate context

more control

participants

rules to take advantage of external services

wizards

familiar devices, but interruptions

34

CarterTechniques and tools for field-based early-stage study and iteration of Ubicomp applications

outline

ubicomp challenges case study

report methods

momento

validation plans

timeline

summary

35

report methods

diary study

experience-sampling method (esm)

CarterTechniques and tools for field-based early-stage study and iteration of Ubicomp applications

36

diary study issues

CarterTechniques and tools for field-based early-stage study and iteration of Ubicomp applications

unobtrusiveness

data sparsity

scale

neglect due to burden of study

generic responses

no feedback

data re-entry, transcribing

37

diary study technique (Carter & Mankoff ’05)

capture

annotate

upload

feedback

elicitation

CarterTechniques and tools for field-based early-stage study and iteration of Ubicomp applications

38

diary study technique: reporter

CarterTechniques and tools for field-based early-stage study and iteration of Ubicomp applications

39

reporter

CarterTechniques and tools for field-based early-stage study and iteration of Ubicomp applications

scale

unobtrusivenessdata sparsity

rapid prototypingany digital capture device

rapid feedback cycle

-

familiar devices

no interruptions

no transcribing

responses tied to captured artifacts feedback not in situ

40

esm: momento

CarterTechniques and tools for field-based early-stage study and iteration of Ubicomp applications

scheduled or context events

automatic question formatting

41

esm: momento

CarterTechniques and tools for field-based early-stage study and iteration of Ubicomp applications

scale

unobtrusivenessdata sparsity

many devices

responses in situ

rapid prototypingknowledge feedback to application behavior

redirect questions

participants

integrate context

familiar devices, but interruptions

42

CarterTechniques and tools for field-based early-stage study and iteration of Ubicomp applications

outline

ubicomp challenges case study

report methods

momento

validation plans

timeline

summary

43

CarterTechniques and tools for field-based early-stage study and iteration of Ubicomp applications

validation: diary study pipeline

testbed: diary study at uc berkeley

interviews with researchers

compare to traditional diary studies (literature)

unobtrusiveness

data sparsity

scale

success?

fewer instances of neglect due to burden of study

more specific recall, fewer generic responses

reduce time to data analysis (no data re-entry)

44

CarterTechniques and tools for field-based early-stage study and iteration of Ubicomp applications

validation: momento esm study

sensor data for awareness app

location, activity data

interviews with wizards, participants

unobtrusiveness

data sparsity

scale

success?

minimal neglect due to burden of study

useful feedback on critical events

wizards able to manage momento queries and complete other work

45

CarterTechniques and tools for field-based early-stage study and iteration of Ubicomp applications

validation: momento scale study

implement dr. blue

wide deployment (>~30)

feedback on critical events, retainment

unobtrusiveness

data sparsity

scale

success?

minimal neglect due to burden of study

useful feedback on critical events

wizards able to manage momento queries and complete other work

rapid prototypingiteration time shorter than similar social mobile apps

46

CarterTechniques and tools for field-based early-stage study and iteration of Ubicomp applications

contributions

a tool (momento) that improves evaluations by simplifying prototyping, making them easier to scale, reducing obtrusiveness, and facilitating data collection

a diary study technique that is easier to scale, less obtrusive, and yields better quality data than previous diary study techniques

identification of major challenges in ubicompevaluation: rapid prototyping, scale, unobtrusiveness, and data sparsity

validation of momento and diary study technique

47

CarterTechniques and tools for field-based early-stage study and iteration of Ubicomp applications

timeline

48

CarterTechniques and tools for field-based early-stage study and iteration of Ubicomp applications

timeline

49

CarterTechniques and tools for field-based early-stage study and iteration of Ubicomp applications

timeline: diary study

50

CarterTechniques and tools for field-based early-stage study and iteration of Ubicomp applications

timeline: momento

51

CarterTechniques and tools for field-based early-stage study and iteration of Ubicomp applications

summary

Leverage pre-existing infrastructures and the increasing power of mobile devices to address the challenges of field evaluations of Ubicomp applications

Wizard-of-Oz and report methods for early stage, field experiments

Ubicomp evaluation challenges: rapid prototyping, scale, unobtrusiveness, data sparsity

52

CarterTechniques and tools for field-based early-stage study and iteration of Ubicomp applications

acknowledgements

Scott Lederer, Christopher Beckmann, Jonathan Hey, Jeffrey Heer, Tara Matthews, Scott Klemmer, James Lin, Tye Rattenbury, Alan Newberger, David Nguyen, Judd Antin, Joe Hall, Nathan Good, Siobhan Hadley

Prof. Robert Kraut, Prof. Susan Fussell, Prof. Jason Hong, Laura Dabbish, Joe Tullio, Jeffrey Nichols, James Fogarty

Techniques and tools for field-based early-stage studyand iteration of Ubicomp applications

53

Scott CarterEECS DepartmentUniversity of California, Berkeleysacarter@cs.berkeley.edu

CommitteeJennifer Mankoff (chair), Anind Dey, John Canny, Peter Lyman

54

CarterTechniques and tools for field-based early-stage study and iteration of Ubicomp applications

other slides

55

CarterTechniques and tools for field-based early-stage study and iteration of Ubicomp applications

worldwide txt stats

number of computer users: about 1 bil by 2006 (IDC)email users: 700 mil by 2006 (Radicati Group)

about 2 bil mobile subscribers by 2006 (In-Stat/MDR)70% use sms (text.it) (number much lower in us)

56

CarterTechniques and tools for field-based early-stage study and iteration of Ubicomp applications

scope

formative and early stages

field methods

distributed, asynchronous activities

>> easier to iterate, could answer some research questions

>> realism especially important in ubicomp >> Sharp & Rehman ‘05: “Real-world deployment is the only way…”>> Kjeldskov: might be able to recreate field in a lab

>> distributed harder, concentrate on asynchronous

broad user base>> minimize programming

57

CarterTechniques and tools for field-based early-stage study and iteration of Ubicomp applications

rapid prototyping

scale

unobtrusiveness

data sparsity

ambiguity

>> simplify interface iteration, reduce coding, reduce burden of reinstallations

>> minimizing infrastructure deployment and time, encourage uptake

>> use familiar devices and lower distraction costs

>> capture and encourage feedback from critical events

>> allow people to control error handling

thesis: addressing challenges

58

CarterTechniques and tools for field-based early-stage study and iteration of Ubicomp applications

approach: wizard of oz

Kelley (1984)

59

CarterTechniques and tools for field-based early-stage study and iteration of Ubicomp applications

approach: wizard of oz

Kelley (1984)

60

CarterTechniques and tools for field-based early-stage study and iteration of Ubicomp applications

approach: wizard of oz

61

CarterTechniques and tools for field-based early-stage study and iteration of Ubicomp applications

approach: report methods

diary

experience sampling

62

CarterTechniques and tools for field-based early-stage study and iteration of Ubicomp applications

approach: report methods

diary

experience sampling>> researchers send questions

63

CarterTechniques and tools for field-based early-stage study and iteration of Ubicomp applications

approach: report methods

diary

experience sampling

>> participants report on events

>> researchers send questions

64

CarterTechniques and tools for field-based early-stage study and iteration of Ubicomp applications

approach: report methods

65

diary studies

feedback or elicitation

participants report on events

form completion (feedback)participant controlled capture (elicitation)

CarterTechniques and tools for field-based early-stage study and iteration of Ubicomp applications

66

advances in recording devices

smaller

part of everyday infrastructure

easier to transfer files

annotation

new, more frequent uses for media

CarterTechniques and tools for field-based early-stage study and iteration of Ubicomp applications

67

findings

+ photos best for who and where recognition

audio elicitation

tangible objects

raw location information

- poor recognition

+/- clandestine captures

- poor recall

+ attitudes and beliefs

- poor sequencing and recall

+ peripheral information often crucial

CarterTechniques and tools for field-based early-stage study and iteration of Ubicomp applications

68

findings

tangible objects and attitudes and beliefs

CarterTechniques and tools for field-based early-stage study and iteration of Ubicomp applications

69

findingsperipheral information often crucial

CarterTechniques and tools for field-based early-stage study and iteration of Ubicomp applications

70

momento

desktop

CarterTechniques and tools for field-based early-stage study and iteration of Ubicomp applications

71

applications

CarterTechniques and tools for field-based early-stage study and iteration of Ubicomp applications

72

applications

interruptibility updates (social, activity)

Momento: How busy are you right now? 1 not busy at all, 5 is really busy.

…

P: Who is most available?

Momento: Joe

CarterTechniques and tools for field-based early-stage study and iteration of Ubicomp applications

73

applications

txt adventure (social, gaming)

Momento: You're in a forest. There is a trail to the N and a cabin to the W. Which way would you like to go? N/W

P: N

Momento: You open the cabin but there was a troll and he squashed you with his fist. Game over. Bad luck

CarterTechniques and tools for field-based early-stage study and iteration of Ubicomp applications

74

applications

near by (location, resource)

CarterTechniques and tools for field-based early-stage study and iteration of Ubicomp applications

75

applications

idea tagger (capture and access)

CarterTechniques and tools for field-based early-stage study and iteration of Ubicomp applications

76

diary study or esm?

CarterTechniques and tools for field-based early-stage study and iteration of Ubicomp applications

77

diary study or esm?

CarterTechniques and tools for field-based early-stage study and iteration of Ubicomp applications

>> auto-driven: participant-sensed events>> creativity, attitudes, beliefs, breadth

diary study

78

diary study or esm?

CarterTechniques and tools for field-based early-stage study and iteration of Ubicomp applications

>> auto-driven: participant-sensed events>> creativity, attitudes, beliefs, breadth

diary study

>> researcher-driven: more control>> time use, depth

esm

79

CarterTechniques and tools for field-based early-stage study and iteration of Ubicomp applications

validation: interface support

augment current application

add functional feature

success?

demonstrate it works

no deployment

novel approach

80

CarterTechniques and tools for field-based early-stage study and iteration of Ubicomp applications

validation: interface support

81

CarterTechniques and tools for field-based early-stage study and iteration of Ubicomp applications

validation: interface support

82

CarterTechniques and tools for field-based early-stage study and iteration of Ubicomp applications

validation: interface support

83

CarterTechniques and tools for field-based early-stage study and iteration of Ubicomp applications

other slides

When Participants Do the Capturing

84

Carter & Mankoff

diary studies and related methods

beeper studies

When Participants Do the Capturing

85

Carter & Mankoff

diary studies and related methods

cultural probes

86

momento: building dr. blue

CarterTechniques and tools for field-based early-stage study and iteration of Ubicomp applications

87

CarterTechniques and tools for field-based early-stage study and iteration of Ubicomp applications

ubicomp evaluation is hard

rapid prototyping

scale

unobtrusiveness

data sparsity

ambiguity

88

CarterTechniques and tools for field-based early-stage study and iteration of Ubicomp applications

case study: dr. blue: ambiguity

89

CarterTechniques and tools for field-based early-stage study and iteration of Ubicomp applications

ubicomp evaluation challenges

rapid prototyping

scale

unobtrusiveness

data sparsity

ambiguity

Metrics: adoption, awareness, distraction, invisibility, (Scholtz and Consolvo)

90

CarterTechniques and tools for field-based early-stage study and iteration of Ubicomp applications

ubicomp challenges case study: hebb

91

CarterTechniques and tools for field-based early-stage study and iteration of Ubicomp applications

case study: hebb: rapid prototyping

92

CarterTechniques and tools for field-based early-stage study and iteration of Ubicomp applications

case study: hebb: rapid prototyping>> simplify interface iteration, reduce coding, reduce burden of reinstallations

93

CarterTechniques and tools for field-based early-stage study and iteration of Ubicomp applications

case study: hebb: scale

94

CarterTechniques and tools for field-based early-stage study and iteration of Ubicomp applications

case study: hebb: scale>> minimize infrastructure deployment and time, encourage uptake

95

CarterTechniques and tools for field-based early-stage study and iteration of Ubicomp applications

case study: hebb: unobtrusiveness

96

CarterTechniques and tools for field-based early-stage study and iteration of Ubicomp applications

case study: hebb: unobtrusiveness>> use familiar devices and lower distraction costs

97

CarterTechniques and tools for field-based early-stage study and iteration of Ubicomp applications

case study: hebb: sparsity

98

CarterTechniques and tools for field-based early-stage study and iteration of Ubicomp applications

case study: hebb: sparsity

>> capture and encourage feedback from critical events

99

CarterTechniques and tools for field-based early-stage study and iteration of Ubicomp applications

case study: hebb: ambiguity

100

CarterTechniques and tools for field-based early-stage study and iteration of Ubicomp applications

case study: hebb: ambiguity

>> allow people to control error handling

101

CarterTechniques and tools for field-based early-stage study and iteration of Ubicomp applications

ubiquitous computing (weiser)

102

CarterTechniques and tools for field-based early-stage study and iteration of Ubicomp applications

ubiquitous computing

off-the-desktop interfaces