[ID] Week 04. Data Collection

Lecture 4

Data Collection

Interface Design/ COM3156, 2015 Fall Class hours : Wedn 1-4 pm 23rd September

SURVEY RESULTS ID Class Team-Up

Lecture #4 COM_Interface Design 2

6 Teams


I II III IV V VI

Team Lead

Project Title

Members

손정민

임신영

이지현

김나래

이주호

김나영

김한솜

김윤성

강단비

변성섭

Dai Yuyu

이민선

박기빈

유화진

박지영

정지원

김하나

진샘

민경진

구연

조예인

송호련

한수교

이현주

DESIGNING FOR THE DIGITAL AGE: HOW TO CREATE HUMAN-CENTERED PRODUCTS AND SERVICES CHAPTER 9. OTHER SOURCES OF INFORMATION AND INSPIRATION

Lecture #4


When You Have Less Time

• When possible, try to squeeze in at least one or two days of user

interviews.

– It's true that you risk getting unusual interview participants who could

skew your thinking, but this is rare, and the risk is limited as long as you

compare what you see to what you're hearing from stakeholders.

– Focus on the kinds of users who are most critical to the product's success

or the ones you suspect are the least understood.

– If you're not allowed to recruit users on your own, use the friends-and-

family method


When You Have Less Time

• A larger interview sample allows for some quantitative analysis of the

data, but is unlikely to result in better design.

• Any existing focus group or survey data may also be useful.

– Don't limit your requests for such information to the product team;

corporate marketing groups frequently conduct research that isn't

disseminated.

– Product managers or professional services staff may also have trip notes

from customer site visits that are worth perusing.

– If possible, review these before the stakeholder meeting so you can use

the data to move things forward.


When You Have More Time

• A larger interview sample allows for some quantitative analysis of the data, which

can be persuasive or comforting to many people.

– However, a larger sample adds considerable analysis time and is unlikely to result in

better design, except to the extent that it lets you catch an edge case you otherwise

might have missed.

• A long session should include an initial interview using the techniques described

in Chapter 7, followed by some mixture of quiet observation time with opportunities

to ask more questions about what you observe.

– If activities cannot be interrupted, use a small video camera; replay interesting

segments for the interviewee later so you can ask questions about specifics. Consider

whether it's better to have the whole team observing or to have just one person follow

the informant with a video camera; surgeons are accustomed to working with large

audiences, but most office workers are not.


Supplemental Research Methods

• Public-space observation


Figure 9.1. If you were designing wayfinding or an information kiosk for an airport, it would be informative to hang out at the information desk for a while.

Exercise

• Would you be comfortable conducting research in the following ways?

Why or why not?

1. You're designing a café it acceptable to buy coffee in a competitive café,

then sit at a table and watch what people do there, how long they stay,

and what seems to encourage them to spend money?

2. You're designing a children's game. Is it acceptable to sit near an

elementary school and watch how children play? How about wandering

over and asking questions of some of the children? What about looking at

their pages on MySpace or another social networking site?


Exercise

3. You're attempting to improve the patient experience at a hospital. Is it acceptable to sit

in a waiting room observing how people behave there?

4. You're designing a patient Web site for a new diabetes drug. Is it acceptable to sit in an

endocrinologist's waiting room and chat with people about their conditions, not

claiming to have diabetes but also not identifying yourself as a researcher?

• Provided you're comfortable doing so, conduct some informal observations in a

public space for the LocalGuide or RoomFinder. What behaviors do you see in this

context that you did not see (or don't think you would have seen) in an interview?

What did you see that provided an interesting insight?



• Mystery shopper

– A variation on public-space observation is being a "mystery shopper," someone

who tries out the customer experience without being identified as a

researcher.

– Anonymous usage is an important research tool because the story you get

during explicit research may be sanitized; this is why good restaurant

reviewers don't tell the staff who they are.

– This technique is a powerful tool for convincing executives of the need for

change; many become passionate advocates for improvement after seeing their own products or services through a customer's eyes. I'm convinced

that more products and services would be better if more executives, as one of

my clients put it, "ate their own dog food."


Exercise

• If you work for a company that sells products or provides customer

support, try using the service as an ordinary customer.

1. Alternatively, try using a service you're not familiar with but might

ordinarily use anyway (such as a physical store, online service, or

telephone customer support center).

2. What makes you feel good about the service and the company providing

it?

3. What gives you a negative opinion?



• Diaries

1. In interviews, it can be difficult to get a sense of behavior over time

because you have to rely on the participant's memory of past activities or

circumstances, and artifacts can only do so much to prompt that.

2. One way to widen your view of someone's activities without shadowing

them 24/7 is to ask them to keep a diary.

3. This can be somewhat structured, much like a survey taken several times,

or can be free-form entry guided by a few questions.



4. A diary can take almost any form: written responses to a periodic e-mail

reminder, a handwritten notebook, a narrated video, or photos with

written commentary.

5. Keep in mind that a diary has limitations: Self-reporting error is likely. If

possible, sit down with each participant and conduct a follow-up

interview using the diary as a basis for questions.



• Surveys

– Numbers helpful in disabusing stakeholders of pet beliefs.

– My opinion alone wouldn't have been persuasive, but quantitative data

helped stakeholders see that they'd be limiting the experience of their

most important audience.


Figure 9.2. An example of a survey with a Likert scale.


• Surveys (continues)

– FINDING EXISTING SURVEY DATA

– DEVELOPING YOUR OWN SURVEY

• Step 1: Identify your audience and goals

• Step 2: Craft questions and instructions

• Step 3: Determine your sample size

• Step 4: Decide how to recruit participants



• Step 1: Identify your audience and goals

– First, work with your teammates (and probably the stakeholders) to

determine what you're trying to learn.

– Is this a single survey to understand demographics and attitudes to inform

interview planning?

– Are you trying to assess how many people fit the behavior patterns you've

already observed?

– Or are you trying to assess the impact of your design with a before and

after comparison?



• Step 2: Craft questions and instructions

– Ask only one question at a time. For example, "If you own a camera, who is

the manufacturer?" is really asking two questions. It would be better to ask

whether the participant owns a camera, then ask who the manufacturer is

only if the first answer is yes.

– Be specific. "Do you ever use the Web?" won't really tell you much, and a

respondent who's used it once may not know how to answer. "How often do

you use the Web? Daily / Two or more times a week / Two or more times a

month / Less than twice a month / Never" would be better. Offer quantity or

frequency choices with specific numbers, since terms like "often" and "seldom"

are relative.



– Make the options for any single-answer question mutually exclusive. For example, if

you're asking about income, don't have a $50,000-$60,000 category and a $60,000-$70,000

category, because the person whose salary is $60,000 won't know what to pick; $50,000-

$59,999 is better.

– Make lists as complete as possible. If someone who drives a minivan has to choose

either "car" or "truck," she will either abandon the survey in frustration or be forced to

choose an answer that doesn't really fit, thereby introducing error.

– Allow for participants who can't provide definitive answers. Include options such as

"other," "not applicable," or "I don't know" when possible, so you don't get incorrect

answers or cause people to quit in frustration.



– Avoid negative construction for multiple choice questions. For

example, "which of the following do you not use" is likely to be read as

"Which of the following do you use." If you absolutely can't avoid it,

visually or verbally emphasize the negative word.

– Limit the options in a Likert scale to five. A scale of 1 to 5 lets people

differentiate between good and great, but a scale of 1 to 10 causes

confusion. Having an odd number of choices allows for a neutral answer,

which generally provides a more accurate picture of attitudes. Make sure

the high/low or positive/negative values are always at the same end of the

scale.



• Use both positive and negative phrasing within your sample. When asking if

people strongly agree or strongly disagree whether something is good, help

minimize bias due to a desire to please by using positive phrasing with half the

group and negative phrasing with the other half. For example, if you present a

Likert scale with a statement like, "Product X is affordable," you will get an

artificially high level of agreement. Balance this by phrasing it as "Product X is

expensive" for the other half of the group.

• Vary list sequences. If possible, vary the sequence of items in lists; this helps

minimize any bias due to people picking things at the top of the list and skipping the

rest. Don't randomize the elements in any list that has a natural progression to it,

such as income or age ranges.



• Step 3: Determine your sample size

– https://www.surveymonkey.com/mp/sample-size-calculator/


Size of population 5% Error 90% Confidence

5% Error 95% Confidence



1,000 214 278 399 648

10,000 265 370 622 1556

100,000 272 383 659 1810

1,000,000 272 384 663 1840

10,000,000 272 384 664 1843

Table 9.1. Sample size examples.

https://www.surveymonkey.com/mp/sample-size-calculator/


• Step 4: Decide how to recruit participants

– Once you've identified how many people you need to recruit, you have to

decide where to find them and how to invite them to participate.

– Many companies recruit users online through invitations on discussion lists,

pop-up invitations on their own Web sites, or e-mail invitations to existing

customers.

– Inviting only your existing customers limits you in two ways. First, it won't tell

you anything about the people you're not reaching. Second, it tends to bias the

results toward favorable responses, since the least happy people have

probably switched to a competitor.

– Inviting only ex-customers provides an unfavorable bias, since they're likely to

be unhappy.



– A mix of current customers, former customers, and those who have never

been customers is the best bet for most studies; proportion depends on the

goal of the survey.

– Inviting people using e-mail, discussion groups, and Web sites is fine if you're

targeting people who spend much time online, but could be missing an

important segment.

– Telephone, advertising, or direct mail may work best in some circumstances.

However, opt-in invitations, such as permanent links on a Web site, may

create a bias toward extreme views, since people who are very happy or very

unhappy are the most likely to seek out those opportunities.

– Opt-out invitations, such as pop-ups and phone calls, minimize this bias.



• Step 5: Decide when and for how long to conduct the survey

– The timing and duration of your survey can have a tremendous impact on your

sample.

– Certain audiences are either more or less available at particular times of year,

days of the week, or times of day.

– If you conduct a telephone survey or in-store study during a weekday, your

results will be skewed toward people who don't work outside the home.

– Even online, some people may be less likely to spend time on a survey during

the work week or during a busy season. Make sure your data collection

window spans enough time to gather data from people with varied habits.


Exercise

• Imagine that the stakeholders for the LocalGuide (see Chapter 6) are

trying to determine how many of a possible ten million business

travelers are likely to use it.

– What questions would help you understand this?

– Develop a survey.

– How large does the sample need to be, assuming a high degree of

confidence is critical?

– How would you recruit?



• Web analytics and customer support data

– When during the day people are visiting your site or particular pages

– How often people visit (if you use identity cookies)

– How long people stay on the site or particular pages

– What percentage of users makes purchases (if applicable)

– Which types of customers are worth the most revenue

– What percentage doesn't purchase the items in their shopping carts

– Where frequent page errors occur

– What sites are referring people to yours



– What terms people entered into a search engine if it's the referring page

– What people search for on your site

– What operating systems and browsers people are using

– The most typical paths through the site (though cached pages can throw

this off)


Figure 9.3. Web analytics can help you identify where there might be a problem, but won't explain why the problem occurs.


• Focus groups

– A focus group is a facilitated, usually 60- to 90-minute meeting with anywhere from five

to a dozen members of a target market.

– The best use for focus groups is when you have a new product idea and know very little

about the people you think might buy and use it.

– If you do plan to conduct a focus group, begin by defining what you want to learn.

• Industry trends?

• General work processes and relationships among roles?

– Consider whether it's more interesting to get the range of views within a particular set of

people or to see how views differ across roles or perspectives. It's generally a good idea

to conduct two to four similar groups; as with interviews, this helps you see if one group

could be an outlier.



• Card sorting

– Card sorting is best suited to relatively small Web sites with users who

understand most of the content. Sorting will help you see that some users

organize mentally based on one criterion while others start from a

different entry point altogether.


Figure 9.4. In card sorting, participants arrange and sometimes edit category names to fit their mental models.


• Competitive products and services

– Companies that spend more time analyzing their competitors than

understanding their customers are likely to be followers rather than

market leaders. That said, a good designer should spend enough time on

the competitors to understand their vulnerabilities and any opportunities

for differentiation.

• Literature and media

– Fiction is also a way for designers to access the zeitgeist that affects how

people respond to new technology products and new interaction

paradigms.


Homework


“Group Homework”

Team Lead & Project Title

1

- Make a Team Blog - The team lead sends me the

URL of the team blog - With a Project Title

Submission Due : 11: 59 pm Sun. 27th September

Education

[ID] Week 04. Data Collection