Scenario-Based Requirements Scenario-Based Requirements for Web Macro Toolsfor Web Macro Tools

Christopher Scaffidi,Allen Cypher, Sebastian Elbaum,Andhy Koesnandar, Brad Myers

22

PBE and browser automationPBE and browser automation

• Programming-by-example (PBE) is a success story– Research systems: Eager, Marquise

– Systems in widespread use: Microsoft Office macros

• Researchers have applied PBE to browser automation– e.g.: Internet Scrapbook, Koala, Miro, Robofox, Turquoise

– These tools…

1. Watch the user perform a task

2. Infer the user’s intent

3. Record a web macro representing the intent

4. Allow the user to replay the macro on new data

introduction scenarios requirements Robofox conclusion

Step 1: Open http://www.oanda.com/converter/classic

Step 2: Submit the amount, date, foreign currency type, and target currency.

Step 3: Retrieve the result in the target currency.

44

PBE web macro tools are PBE web macro tools are not widely adoptednot widely adopted

• The web context introduces new challenges to PBE:– Web site changes can break macros without warning

Need features for detecting and responding to site change

– Office tasks involve not just browsersbut also spreadsheets and other applications Need integration with other applications

• What other unmet requirements are there?

introduction scenarios requirements Robofox conclusion

55

Missing: a methodical characterization of Missing: a methodical characterization of requirements for web macro toolsrequirements for web macro tools

• Many papers use scenarios to explain and motivate a PBE tool’s features.

• Such scenarios…– Are retrospective: The scenarios show the features that

actually work, rather than opportunities for future work.

– Lack pedigree: Are the scenarios hypothetical or based on a user study? Will the tool features be useful in practice?

introduction scenarios requirements Robofox conclusion

66

OutlineOutline

• We have…– Analyzed 10 scenarios where web macros would be desirable

– Characterized 25 requirements that tools must support to be useful for many real-world tasks

– Used these requirements as a benchmark to identify opportunities for enhancing a particular tool (Robofox)

introduction scenarios requirements Robofox conclusion

77

Catalog of scenariosCatalog of scenarios

• Using a web site to look up information – 6 scenarios– Currency converter, Package tracker, Per diem lookup,

Staff lookup, Stock analysis, Watcher for eBay

– Typical users: office workers

• Using a web form to order products – 1 scenario– Path to procurement

– Typical users: online shoppers

• “Scraping” a large part of a site – 3 scenarios– Peoplesoft scraper, Person locator scraper,

Scraper for CMS

– Typical users: System administrators

introduction scenarios requirements Robofox conclusion

88

Scenarios came from a variety of sourcesScenarios came from a variety of sources

• Contextual inquiry – 3 scenarios– Currency converter, Per diem lookup, Staff lookup

• Co-workers – 2 scenarios– Path to procurement, Peoplesoft scraper

• Our own experiences – 2 scenarios– Package tracker, Scraper for CMS

• Online sources – 3 scenarios– Person locator scraper, Stock analysis, Watcher for eBay

introduction scenarios requirements Robofox conclusion

99

What is in a scenario?What is in a scenario?

* Title Pithy name to make it easy to refer to the scenario

* Typical User Tells what type of person is likely to perform the task

* Overview Context and forces that would prompt a user to do the task

* Starting Conditions Pre-conditions that hold true before the scenario

* Result Post-conditions that should hold true after the scenario

* Actions Steps that the user performs (maybe with a tool)

Action Details Screenshots or other clarifying information

Variations Discusses likely tweaks of the main scenario

Macro Maintenance

Examines how scenario evolution prompting macro changes

* Scenario Source Summarizes empirical data that generated this scenario

* = required section introduction scenarios requirements Robofox conclusion

1010

Requirements: Triggering macrosRequirements: Triggering macros

• On-demand execution– Required to support users’ work tasks

• Scheduled execution– Especially for scrapers

• Event-based triggers– Watcher for eBay executes when user visits Google

homepage

• Subroutines– Especially for lookup macros

introduction scenarios requirements Robofox conclusion

1111

Requirements: Using objects on pagesRequirements: Using objects on pages

• Text snippets– Almost all scenarios require reading text

• Tabular information -– Almost all scenarios read tabular data

• Web form widgets– Read/write textbox, dropdown, checkbox widgets

• Other HTML structures– Watcher for eBay

introduction scenarios requirements Robofox conclusion

1212

Requirements: Adapting to site changesRequirements: Adapting to site changes

• Adaptation to changing page layout– Changes to text location, text color, page structure

• Adaptation to changing form fields -– Changes to field names and field values

• Adaptation to changing URLs– “This page has moved to X. Please update your links.”

introduction scenarios requirements Robofox conclusion

1313

Requirements: Reading and writing dataRequirements: Reading and writing dataoutside pagesoutside pages

• Browser APIs– Read/write highlighted text, show popup window

• Spreadsheets and other files -– Often one record per spreadsheet row or XML node

• Parameters containing user input– Prompting for an input (e.g. password) during playback

introduction scenarios requirements Robofox conclusion

1414

Requirements: Transforming dataRequirements: Transforming data

• Reformat to equivalent value -– For comparing values and for making outputs consistent

• Extracting values’ parts– For comparing values

• Combining values– Such as arithmetic and string concatenation

introduction scenarios requirements Robofox conclusion

1515

Requirements: Executing control structuresRequirements: Executing control structures

• Looping operations -– Especially in scrapers

• Conditional operations– Macro behavior depends on data provided by user as well

as data returned by web sites

introduction scenarios requirements Robofox conclusion

1616

Requirements: Recovering from failureRequirements: Recovering from failure

• Partial restarts– Especially for scrapers

• Exception handlers– Integration with assertions

introduction scenarios requirements Robofox conclusion

1717

Requirements: Supporting macro Requirements: Supporting macro maintenancemaintenance

• User-understandable representation– For maintenance and for sharing macros with other users

• Editable macros– Analogous to features supported by textual editors

• Features for debugging– Traces, breakpoints, step-by-step execution, etc

• Maintenance at runtime– On error, enter maintenance mode

introduction scenarios requirements Robofox conclusion

1818

Example benchmark use: RobofoxExample benchmark use: RobofoxTriggering macros

On-demand execution Yes

Scheduled execution Yes

Event-based triggers Yes

Subroutines No

Using objects on web pages

Text snippets Yes

Tabular information No

Web form widgets Yes

Other HTML structures No

Adapting to site changes

Adaptation to changing page layout Limited

Adaptation to changing form fields Yes

Adaptation to changing URLs No

• Inserts “sanity checks” to detect changing page structure.

• Includes heuristics for “refinding” page elements based on color, label, DOM location.

• Some changes (e.g.: form field renaming) are not handled automatically.

introduction scenarios requirements Robofox conclusion

1919

Example benchmark use: RobofoxExample benchmark use: Robofox

• Can read/write spreadsheets, but only if they have a specific layout

Reading and writing data outside pages

Browser APIs Limited

Spreadsheets and other files Limited

Parameters containing user input Limited

Transforming data

Reformat to equivalent value No

Extracting values’ parts Yes

Combining values No

Executing control structures

Looping operations Limited

Conditional operations Yes

• Can do a “foreach” loop but not a “while(condition)” loop

introduction scenarios requirements Robofox conclusion

2020

Example benchmark use: RobofoxExample benchmark use: Robofox

• User-defined exception handlers can fire when assertions fail

Recovering from failure

Partial restarts No

Exception handlers Yes

Supporting macro maintenance

User-understandable representation Yes

Editable macros Yes

Features for debugging Yes

Maintenance at runtime Yes

introduction scenarios requirements Robofox conclusion

2121

Summary of Robofox limitationsSummary of Robofox limitations

• Of 25 requirements…– Supported: 13

– Limited support: 5

– Unsupported: 7

• Of 10 scenarios…– Supported: 3

– Limited support: 2 (unsupported scenario variations)

– Unsupported: 5

introduction scenarios requirements Robofox conclusion

2222

Opportunities to enhance RobofoxOpportunities to enhance Robofox

• Adding support for 2 requirements…– Tabular information

– Reformat to equivalent value

• … would leave only…– 2 unsupported scenarios

– 2 unsupported scenario variations

introduction scenarios requirements Robofox conclusion

2323

Extending the benchmarkExtending the benchmark

• These 25 requirements are “Version 1” of a benchmark for evaluating web macro tools.

• As researchers meet existing requirements, new scenarios and new requirements will appear.

• We created a wiki where YOU can upload scenarios.

– http://softwaresurvey.cs.cmu.edu/wmcorpus.html

introduction scenarios requirements Robofox conclusion

2424

Thank You…Thank You…

• To VL/HCC for the opportunity to present.

• To EUSES Consortium members for suggestions.

• To NSF and EUSES for funding(CCF-0325273, CCF-0438929, CCF-0324861).

introduction scenarios requirements Robofox conclusion