IAT 334 Experimental Evaluation ______________________________________________________________________________________ SCHOOL OF INTERACTIVE ARTS + TECHNOLOGY [SIAT] | WWW.SIAT.SFU.CA
March 10, 2011IAT 3342 Evaluation Evaluation styles Subjective data Questionnaires, Interviews Objective data Observing Users Techniques, Recording Usability Specifications Why, How
March 10, 2011IAT 3343 Our goal?
March 10, 2011IAT 3344 Evaluation Earlier: Interpretive and Predictive Heuristic evaluation, walkthroughs, ethnography Now: User involved Usage observations, experiments, interviews...
March 10, 2011IAT 3345 Evaluation Forms Summative After a system has been finished. Make judgments about final item. Formative As project is forming. All through the lifecycle. Early, continuous.
March 10, 2011IAT 3346 Evaluation Data Gathering Design the experiment to collect the data to test the hypothesis to evaluate the interface to refine the design Information we gather about an interface can be subjective or objective Information also can be qualitative or quantitative Which are tougher to measure?
March 10, 2011IAT 3347 Subjective Data Satisfaction is an important factor in performance over time Learning what people prefer is valuable data to gather
March 10, 2011IAT 3348 Methods Ways of gathering subjective data Questionnaires Interviews Booths (eg, trade show) Call-in product hot-line Field support workers
March 10, 2011IAT 3349 Questionnaires Preparation is expensive, but administration is cheap Oral vs. written Oral advs: Can ask follow-up questions Oral disadvs: Costly, time-consuming Forms can provide better quantitative data
March 10, 2011IAT 33410 Questionnaires Issues Only as good as questions you ask Establish purpose of questionnaire Dont ask things that you will not use Who is your audience? How do you deliver and collect questionnaire?
March 10, 2011IAT 33411 Questionnaire Topic Can gather demographic data and data about the interface being studied Demographic data: Age, gender Task expertise Motivation Frequency of use Education/literacy
March 10, 2011IAT 33412 Interface Data Can gather data about screen graphic design terminology capabilities learning overall impression ...
March 10, 2011IAT 33413 Question Format Closed format Answer restricted to a set of choices Characters on screen hard to read easy to read 1 2 3 4 5 6 7
March 10, 2011IAT 33414 Closed Format Likert Scale Typical scale uses 5, 7 or 9 choices Above that is hard to discern Doing an odd number gives the neutral choice in the middle
March 10, 2011IAT 33415 Closed Format Advantages Clarify alternatives Easily quantifiable Eliminates useless answers Disadvantages Must cover whole range All should be equally likely Dont get interesting, different reactions
March 10, 2011IAT 33416 Issues Question specificity Do you have a computer? Language Beware terminology, jargon Clarity Leading questions Can be phrased either positive or negative
March 10, 2011IAT 33417 Issues Prestige bias People answer a certain way because they want you to think that way about them Embarrassing questions Hypothetical questions Halo effect When estimate of one feature affects estimate of another (eg, intelligence/looks)
March 10, 2011IAT 33418 Deployment Steps Discuss questions among team Administer verbally/written to a few people (pilot). Verbally query about thoughts on questions Administer final test
March 10, 2011IAT 33419 Open-ended Questions Asks for unprompted opinions Good for general, subjective information, but difficult to analyze rigorously May help with design ideas Can you suggest improvements to this interface?
March 10, 2011IAT 33420 Ethics People can be sensitive about this process and issues Make sure they know you are testing software, not them Attribution theory Studies why people believe that they succeeded or failed--themselves or outside factors (gender, age differences) Can quit anytime
March 10, 2011IAT 33421 Objective Data Users interact with interface You observe, monitor, calculate, examine, measure, Objective, scientific data gathering Comparison to interpretive/predictive evaluation
March 10, 2011IAT 33422 Observing Users Not as easy as you think One of the best ways to gather feedback about your interface Watch, listen and learn as a person interacts with your system
March 10, 2011IAT 33423 Observation Direct In same room Can be intrusive Users aware of your presence Only see it one time May use semitransparent mirror to reduce intrusiveness Indirect Video recording Reduces intrusiveness, but doesnt eliminate it Cameras focused on screen, face & keyboard Gives archival record, but can spend a lot of time reviewing it
March 10, 2011IAT 33424 Location Observations may be In lab - Maybe a specially built usability lab Easier to control Can have user complete set of tasks In field Watch their everyday actions More realistic Harder to control other factors
March 10, 2011IAT 33425 Challenge In simple observation, you observe actions but dont know whats going on in their head Often utilize some form of verbal protocol where users describe their thoughts
March 10, 2011IAT 33426 Verbal Protocol One technique: Think-aloud User describes verbally what s/he is thinking and doing What they believe is happening Why they take an action What they are trying to do
March 10, 2011IAT 33427 Think Aloud Very widely used, useful technique Allows you to understand users thought processes better Potential problems: Can be awkward for participant Thinking aloud can modify way user performs task
March 10, 2011IAT 33428 Teams Another technique: Co-discovery learning Join pairs of participants to work together Use think aloud Perhaps have one person be semi-expert (coach) and one be novice More natural (like conversation) so removes some awkwardness of individual think aloud
March 10, 2011IAT 33429 Alternative What if thinking aloud during session will be too disruptive? Can use post-event protocol User performs session, then watches video afterwards and describes what s/he was thinking Sometimes difficult to recall
March 10, 2011IAT 33430 Historical Record In observing users, how do you capture events in the session for later analysis?
March 10, 2011IAT 33431 Capturing a Session 1. Paper & pencil Can be slow May miss things Is definitely cheap and easy Time 10:00 10:03 10:08 10:22 Task 1 Task 2 Task 3 SeSe SeSe
March 10, 2011IAT 33432 Capturing a Session 2. Audio tape Good for talk-aloud Hard to tie to interface 3. Video tape Multiple cameras probably needed Good record Can be intrusive
March 10, 2011IAT 33433 Capturing a Session 4. Software logging Modify software to log user actions Can give time-stamped key press or mouse event Two problems: Too low-level, want higher level events Massive amount of data, need analysis tools
March 10, 2011IAT 33434 Assessing Usability Usability Specifications Quantitative usability goals, used a guide for knowing when interface is good enough Should be established as early as possible in development process
March 10, 2011IAT 33435 Measurement Process If you cant measure it, you cant manage it Need to keep gathering data on each iterative refinement
March 10, 2011IAT 33436 What to Measure? Usability attributes Initial performance Long-term performance Learnability Retainability Advanced feature usage First impression Long-term user satisfaction
March 10, 2011IAT 33437 How to Measure? Benchmark Task Specific, clearly stated task for users to carry out Example: Calendar manager Schedule an appointment with Prof. Smith for next Thursday at 3pm. Users perform these under a variety of conditions and you measure performance
March 10, 2011IAT 33438 Assessment Technique Usability Measure Value to Current Worst Planned Best poss Observ attribute instrument be measured level acc level target level level results Initial Benchmk Length of 15 secs 30 secs 20 secs 10 secs perf task time to (manual) success add appt on first trial First Quest -2..2 ?? 0 0.75 1.5 impression
March 10, 2011IAT 33439 Summary Measuring Instrument Questionnaires Benchmark tasks
March 10, 2011IAT 33440 Summary Value to be measured Time to complete task Number of percentage of errors