SWET‐QUM: A Quality in Use Extension Model for Semantic Web Exploration Tools

SWET-‐QUM: A Quality in Use Extension Model for

Seman;c Web Explora;on Tools

José Luis González, Roberto García, Josep Maria BruneI, Rosa Gil and Juan Manuel Gimeno

Universitat de Lleida

Introduc;on •  Objec;ve:

•  Define a model to evaluate the quality of applica;ons based on Seman;c Web Technologies

•  Focus on: •  Quality in Use •  Seman3c Web Explora3on Tools (SWETs)

2 SWET-‐QUM: A Quality in Use Extension Model for Seman;c Web Explora;on Tools

Quality of a SoVware Product


Semantic Datasets

http://linkeddata.org

295 datasets


http://linkeddata.org

Fine for computers… but people?

M. Harper (developer)

I’m developing a bird watching application. Can I reuse DBPedia?

C. Warren (blogger)

I’m writing about “Films I Like”. Can I reuse LinkedMDB?


Mo;va;on •  Seman3c Web HCI Challenges:

•  Lot of data but... available as RDF data dumps, seman;c query services, etc

•  It requires Seman3c Web exper3se •  Exis;ng tools hardly usable for lay users

User Interface needed


SWET-‐QUM •  Extension of ISO/IEC 25010:2011:

•  Metrics based on standards

+ •  Metrics specially conceived for SWETs


Quality in Use Model


Effec;veness •  Defini3on: the degree to which users can achieve the seman2c data explora2on tasks with precision and completeness

•  Metrics: •  Task success: what % of one task is completed? •  Task comple3on: what % of the tasks is completed? Extension •  UI Component Effec3veness: what % of the UI components, relevant for the task, do the users view?

•  Input: eye tracking record of screen areas


Efficiency •  Defini3on: the degree to which users can achieve tasks by

inves2ng an appropriate amount of resources in rela2on to the effec2veness

•  Metrics: •  Task 3me: how long does it take to complete a task? •  Total 3me: how long does it take to complete all the tasks? •  Task efficiency: task success / task ;me •  Help requests: how many help requests has the user asked?

Extension •  UI Component Efficiency: what % of the ahen;on of the user is captured by the UI components relevant for the task?

•  Input: eye tracking record of screen areas


Sa;sfac;on •  Defini3on: the degree to which users are sa2sfied by the seman2c data explora2on tool

•  Metrics: •  Sa3sfac3on ques3onnaire: how sa;sfied is the user with specific soVware features?


Context Coverage •  Defini3on: the degree to which Seman2c Web tools can be used in different contexts and adapt to different user mental models

•  Metrics: Extension •  Task Flexibility: what % of the task can be achieved by alterna;ve methods of interac;on?

•  Layout Flexibility: what is the average number of interac;on steps required to reach the UI Components?

•  Input: expert analysis of the tasks, applica;on domain and user interface.


Case Study •  Rhizomer: tool for Seman;c Data explora;on

•  Automa3c and generic UI: menus, facets, breadcrumbs… •  Methodology:

•  User-‐centered design development process •  RITE Method: short itera;ons, more evalua;ons, small groups

•  SWET-‐QUM model •  Evalua3on proces:

•  Pre-‐test •  Test •  Post-‐test •  Reports


Case Study •  Dataset: LinkedMDB (seman;c version of IMDB.com)

•  Tasks: •  Task A: “Find three films where Clint Eastwood is director and also actor”

•  Task B: “Find who has directed more films in countries located in Oceania”


Results •  Effec3veness and efficiency:

15

Task Task Success Task Time (min)

Help Requests Task Efficiency

Task A 100 % 1.69 0 % 68 %

Task B 100 % 3.43 100 % 32 %

SWET-‐QUM: A Quality in Use Extension Model for Seman;c Web Explora;on Tools

Results •  UI Components Effec3veness and Efficiency:

16

UI Components Fixa3on count

Fixa3on Dura3on (s)

UI Component Effec3veness

UI Component Efficiency

Naviga3on bar 14 4.1 100 % 4 %

Facets 161 91.92 100 % 90 %

Facets “pivot bu[on” 5 3.75 100 % 4 %

“Navigate to” Box 3 1.15 100 % 1 %

Breadcrumbs 7 1.78 100 % 2 %

Total 190 102.7 -‐ 100 %


Heat Map


Results •  Context coverage:

18

Task A Task B

Task Flexibility 100 % It is possible to go through actor, director or film

50 % It is possible to go throuh director or film, but not

through country or con;nent

Layout Flexibility 1.5 Zoom film (0), expand facet (1), search box (2), select person (3), expand facet (1), select person (2)

1.83 Zoom film (0), pivot country (1), expand con;nent (2),

select Oceania (3), pivot film (2), expand director (3)

Interac;on Steps 6 6


Results •  Sa3sfac3on: (from 1 to 5)

•  S-‐G1 The task was... very hard -‐ very easy •  S-‐G2 I think that I have done the task... not correctly at all -‐absolutely correct

•  S-‐G3 The interface structure... did not help me at all -‐ did help me very much

•  S-‐G4 The ;me to complete task has been... long -‐ short •  S-‐G5 To achieve the task I have had to be... very focused – not focused at all

•  S-‐G6 The task was... badly defined, I did not understand the objec;ve -‐ Well defined, I understood the objec;ve


Results •  Sa3sfac3on:

Task A Task B


Conclusions and Future Work •  Conclusions:

•  Model to evaluate the Quality in Use of Seman;c Web Explora;on Tools

•  Standard metrics + SW metrics •  Applied in Rhizomer à maximise the quality of user experience

•  Future Work: •  Explore addi3onal metrics •  Leverage eye tracking •  Apply to other tools


Thanks for your attention

Josep Maria Brunetti [email protected]

http://rhizomik.net


Technology

SWET‐QUM: A Quality in Use Extension Model for Semantic Web Exploration Tools