View
0
Download
0
Category
Preview:
Citation preview
A Physical Mobile Interactions Framework based on Semantic DescriptionsSven Siorpaes, Diploma Thesis
Duration: 01.01.2006 – 30.06.2006Supervisors: Dipl.-Inf. Enrico Rukzio, Massimo Paolucci (NTT DoCoMo Euro-Labs), John Hamard NTT DoCoMo Euro-Labs)Responsible Lecturer: Dr. Albrecht SchmidtResponsible Professor: Prof. Dr. Heinrich Hußmann
Mobile Interaction withReal World ServicesGregor Broll, Diploma Thesis
Duration: 01.01.2006 – 30.06.2006Supervisors: Dipl.-Inf. Enrico Rukzio, Massimo Paolucci (NTT DoCoMo Euro-Labs), John Hamard NTT DoCoMo Euro-Labs)Responsible Lecturer: Dr. Albrecht SchmidtResponsible Professor: Prof. Dr. Heinrich Hußmann
Sven Siorpaes, Gregor Broll, 26.07.2006 3/48
Outline
• Introduction, Challenge and Approach
• Selected Issues of Related Work
• Use-Case Scenarios and Paper Prototyping
• Framework Architecture
• System Descriptions Supporting the Interaction Process
• Focus: User Interface Generation Process
• Prototype Client Implementation
• User Study and Prototype Evaluation
• Conclusion and Future Work
Sven Siorpaes, Gregor Broll, 26.07.2006 4/48
Introduction, Challengeand Approach
Sven Siorpaes, Gregor Broll, 26.07.2006 5/48
Introduction
• Increasing interest in Physical Mobile Interaction
• Facilitates mobile interaction with digital services through the interaction with physical objects
• Powerful mobile devices for information access, collection, processing and interaction
• (Augmented) physical objects become recognizable
• Techniques: pointing, touching, scanning, location, …
• Technologies: visual marker and pattern recognition, wireless RFID / NFC tags, laser pointer, Bluetooth, GPS, …
• Objects get digital identities ( Internet of things) and can be associated with information and services
Sven Siorpaes, Gregor Broll, 26.07.2006 6/48
Challenge
• Current implementations of Physical Mobile Interactions mostly simple and proprietary prototypes
• Limited scope of application and mostly single interaction techniques
• Little tool- and framework-support
Support more complex Physical Mobile Interactions e.g. represent a sequence of interactionsShift focus of interaction from mobile devices onto physical objects (e.g. from the Internet of Things)Transfer the familiarity of interacting with physical objects and exploit it for more intuitive interaction with associated servicesProvide an independent service infrastructure that is reusable across different services and interactionsAutomatic user interface generation required that abstracts fromspecific target platform
Sven Siorpaes, Gregor Broll, 26.07.2006 7/48
Approach
• Perci project (PERvasive ServiCe Interaction), a collaboration between LMU and DoCoMo Eurolabs
• Goal: Combination of Physical Mobile Interaction and Semantic Web Services for their mutual benefit
• Generic framework to exploit the expressiveness, flexibility and interoperability of Semantic Web Services for richer Physical Mobile Interactions
• Use extended Web Service descriptions for the automatic generation of adapted interfaces that support and facilitate Physical Mobile Interaction
Sven Siorpaes, Gregor Broll, 26.07.2006 8/48
MobileDeviceService Semantic Web
ServiceConcrete
User InterfaceAbstract
User Interface
Context
First Framework Outline
Sven Siorpaes Gregor Broll
Sven Siorpaes, Gregor Broll, 26.07.2006 9/48
Selected Issues of Related Work
Sven Siorpaes, Gregor Broll, 26.07.2006 10/48
Web Services and Semantic Web Services
Web Services• Standardized Web Service Description Language (WSDL) to specify service
invocation interface• Interoperation between heterogeneous platforms, e.g. hardware or software
platforms• Well established standard in industry and academia, extending existing
Web Services like Amazon or Ebay
Semantic Web Services• Standardized service description ontology OWL-S• Connect atomic WSDL operations to complex processes, e.g. sequence• Provides semantic expressiveness by adding an abstract type system to the
syntactic WSDL message formats
Sven Siorpaes, Gregor Broll, 26.07.2006 11/48
User Interface DescriptionLanguages (UIDL)
• UIDLs are the first step for the automatic generation of interfaces
• Facilitate development of application interfaces for different platforms; reusable, easy to learn, more effective prototyping, ...
• Scope of Perci: first step of the transition from Semantic Web Service descriptions to interfaces for Physical Mobile Interaction
• Evaluation of different abstract and concrete UIDLs: UIML, XUL, USIXML, XAML, WSXL, markup languages (XHTML, WML, cHTML)
• Mostly not suitable for the Perci approach: too inflexible or not genericenough (UIML), too concrete and heavyweight (XUL, XAML) or too general(USIXML), not enough support for mobile interfaces, no connection to Web Services
• Evaluated UIDLs only as drafts for the creation of an own abstract userinterface description language
Sven Siorpaes, Gregor Broll, 26.07.2006 12/48
Automatic Generation of User Interfaces
• Intermediates between SWS interface descriptions and Physical Mobile Interaction on mobile devices
• SUPPLE: Framework for automatic generation of UIs- Based on models for devices, users and abstract functional interface specification
- Algorithm maps widgets to suitable UI elements based on efficiency constraints
• Pebbles: using mobile devices for controlling electronic appliances- Interface generation based on abstract descriptions of appliances and their functions
- support for different modalities, device constraints and user preferences
- 2-way communication for sending messages using device adaptors
Sven Siorpaes, Gregor Broll, 26.07.2006 13/48
Ontological Approach to Generating UIs for Web Services
• D. Khushraj and O. Lassila: Architecture for the automatic creation and personalisation of dynamic UI forms from OWL-S Profile and Process Modelof Semantic Web Services
• OWL-S ontology to describe UIModel for SWS
• Extension of this ontology with user interface annotations(e.g. display labels)
• Exploiting relationship between WS input and association with info about a user from Semantic Cache for personalisation
• Role model for own framework for Physical Mobile Interactions
Sven Siorpaes, Gregor Broll, 26.07.2006 14/48
The Internet of Things
• Industrial effort to wirelessly tag and identify physical objects, e.g. using RFID
• Everyday objects get individual digital identities and references
• Objects can be presented, identified and linked with each other by means of a network infrastructure => “Internet of Things”
• Increasing importance in industry; automatic recognition, identification, tracking and monitoring of products
• Standardization of framework for identification and description driven by EPCGlobal and Auto-ID Labs.
• Infrastructure components: RFID-tags,Electronic Product Code (EPC) for unique identification, Object Naming Service (ONS) for matching EPC and PML and Physical Markup Language (PML) for describing object properties
Sven Siorpaes, Gregor Broll, 26.07.2006 15/48
Use-Case Scenarios and Paper Prototyping
Sven Siorpaes, Gregor Broll, 26.07.2006 16/48
Use-Case Scenarios
Sven Siorpaes, Gregor Broll, 26.07.2006 17/48
Paper-Prototyping and First User Study
• Simple user study with 10 participants (mostly students)
• Complete 2 scenarios with the posters and the paper prototypes (buying a movie ticket and a transportation ticket)
• Questions about the system before and after the scenarios
Sven Siorpaes, Gregor Broll, 26.07.2006 18/48
Early User Study - Results
• 70% of the users think that the proposed system is useful
• Initial effort to understand the system but then easy and intuitive to use, if users are already familiar with a mobile phone
• Useful where poster replaces another automat, but in some cases users could prefer a human contact for feedback (e. g. ticket counter)
+ Fast, low-cost, can be used anywhere, easy to replace
+ Less complicated menus, easy physical interaction, less faults
+ Added value: payment could be included into mobile phone
- NFC widely unknown, needs to be established
- Not enough feedback, only from mobile; actions not reversible
- Posters need to be put up and actualized
Sven Siorpaes, Gregor Broll, 26.07.2006 19/48
Redesign I
• Added support for visual marker recognition and direct input
• Incorporated the suggestions from the user study
• Added more hints and instructions on how to use the posters
• Definition of action/task-tags and parameter/option tags in order to make the poster‘s functionality more modular and extendible
• Transportation poster will use a new model for choosing stations more efficiently
Sven Siorpaes, Gregor Broll, 26.07.2006 20/48
Redesign II
Sven Siorpaes, Gregor Broll, 26.07.2006 21/48
FrameworkArchitecture
Sven Siorpaes, Gregor Broll, 26.07.2006 22/48
Overview
• Architecture bridging the gap between the Web Service Domain and the Physical Mobile Interaction Domain
• A Universal Client running on a mobile device is interacting with Physical Objects and providing a technical connection to services
• Component Interaction Proxy (IAProxy) that mediates between the two domains
Service 1 Universal Clienton mobile device
Physical Object
Web ServiceDomain
Physical MobileInteraction Domain
Domain Gap
...
Corresponds to
InteractionProxyService 2
Service n
Sven Siorpaes, Gregor Broll, 26.07.2006 23/48
Services
• Services provide different descriptions to allow controlling the interaction flow and the automatic user interface generation:
- Functional behaviour and abstract parameter types of the service, defined as Service Description standards WSDL and OWL-S
- Own user interface relevant Service Annotations based on OWL
Sven Siorpaes, Gregor Broll, 26.07.2006 24/48
Interaction Proxy
• Interaction Proxy responsibilities and components: - Interaction management (Service Connector)
- Automatic user interface generation (UI Generator)
- Interface provision for the Universal Client (Client Connector)
Sven Siorpaes, Gregor Broll, 26.07.2006 25/48
Universal Client and Physical Object
• Universal Client:- Service Client for a bidirect communication with IAProxy
- Interaction Client reads data from Physical Objects
• Physical Objects store a type and value corresponding to the service
Sven Siorpaes, Gregor Broll, 26.07.2006 26/48
Interaction Sequence
Service Interaction Proxy
Request initial user interface description
Read Action Tag
Read service descriptions and annotation
Respond user interface description
Physical object
Generate user interface description
Universal Client
Read Parameter Tags
...Submit service invocation request
Invoke service
Return service output
Generate user interface description including service output
Respond user interface description
...
Sven Siorpaes, Gregor Broll, 26.07.2006 27/48
System DescriptionsSupporting the
Interaction Process
Sven Siorpaes, Gregor Broll, 26.07.2006 28/48
System Descriptions I
OWL-S:
• Models complex interactions
• Not sufficient for UI generation
Sven Siorpaes, Gregor Broll, 26.07.2006 29/48
System Descriptions II
Service User Interface Annotation (SUIA): • Human readable labels and descriptions
for service parameters
• Reference to abstract widget types
• Reference to predefined value set for the parameter
Sven Siorpaes, Gregor Broll, 26.07.2006 30/48
System Descriptions III
Abstract UI description: • Contains all information for generating
and rendering the user interface
• Direct interpretation by device oradditional transformation to targetlanguage
Sven Siorpaes, Gregor Broll, 26.07.2006 31/48
Focus: User InterfaceGeneration Process
Sven Siorpaes, Gregor Broll, 26.07.2006 32/48
User Interface Generation Process
XSLT Transformation of different description sources to composed Abstract UI Description
Two target platforms depending on the Device profile:• Direct interpretation on J2ME runtime
• Additional transformation to create a representation for an XHTML browser
OWL-S description
SUIA description
Service output
XSLT Transformation
Web Service Interaction Proxy
Abstract UI Description
Universal Client
Device profile(J2ME or XHTML)
J2ME runtime
XHTML browserXSLT Transformation
[XHTML platform]
[J2ME platform]
XHTML
Apache Cocoon Framework
Abstract UI to XHTML stylesheet
Abstract UI generation stylesheet
Abstract Parameter Types
Abstract Widget Type Model
Sven Siorpaes, Gregor Broll, 26.07.2006 33/48
Prototype Client Implementation
Sven Siorpaes, Gregor Broll, 26.07.2006 34/48
Prototype Overview
Sven Siorpaes, Gregor Broll, 26.07.2006 35/48
J2ME Prototype Implementation
• Prototype implemented with J2ME (CLDC 1.0/1.1 and MIDP 2.0), the Nokia RFID & NFC SDK 1.0 and kXML
• Uses NFC and visual marker recognitionthrough PMIF (Physical Mobile Interaction Framework)
• No mobile device with technical outfit to support all technologies at the same time
• Development and testing with Nokia 3220 (plus NFC shell) and 6630 mobile phones
• Posters were augmented with NFC-tags and visual markers from www.visualcode.org
Sven Siorpaes, Gregor Broll, 26.07.2006 36/48
Supported Interaction Techniques
Touching
Pointing
Direct Input
Sven Siorpaes, Gregor Broll, 26.07.2006 37/48
Generic Components
• InteractionClient:Encapsulates and manages different PMIF interactiontechnologies and techniques that provide interpretation of marker based data
• PerciClientMidlet:Manages the application logic through an update mechanism that handles information acquired throughPhysical Mobile Interaction
• ServiceClient:Manages connection to the Interaction Proxy and theretrieval of interface descriptions according to user/devicecontext
• GenericForm:Interface container; generates concrete widgets fromAbstractUIDescriptions, manages the interaction with themand updates them upon Physical Mobile Interaction
Sven Siorpaes, Gregor Broll, 26.07.2006 38/48
Generic Component Interaction
Interaction Client
PerciClient Midlet
GenericForm
Service Client
IA Proxy
PMI
data
manage() update()
XML interface
description
PTgenerateand update interface
service URL /parameters
XML interfacedescription
connect to service
AT
Sven Siorpaes, Gregor Broll, 26.07.2006 39/48
Interface Generation I
• AbstractUIDescription is parsed into an UIObject as the basis for theinterface generation
• The Abstract Widget Type Model is used to map different widget-descriptions to concrete J2ME widgets
• Mapping is decided by the client and its context (e.g. support for different interaction techniques or user preferences)
• Widgets are created anonymously and have to be stored and registered forfuture reference (e.g. for updating widgets or reading their values for serviceinvocation)
Sven Siorpaes, Gregor Broll, 26.07.2006 40/48
Interface Generation II
Sven Siorpaes, Gregor Broll, 26.07.2006 41/48
Physical Mobile Interaction /Updating Interfaces
<tag type="parameter"><abstractType>http://perci.medien.ifi.lmu.de:8080/axis/domain/cinema/cinema.owl#MovieTitle</abstractType><value>XMen 3</value><label>XMen 3</label><desc>The X-Men make a last stand in the war between humans and mutants.</desc>
</tag>
Matching AbstractParameter Types
Sven Siorpaes, Gregor Broll, 26.07.2006 42/48
User Study and Prototype Evaluation
Sven Siorpaes, Gregor Broll, 26.07.2006 43/48
User Study and Evaluation I
• 10 participants, aged from 23 to 46 (average 30.7), 8 participants with technical background
• Process - User-Experiment: Accomplish to buy cinema tickets for given properties.
Use all interaction techniques (Touching, Pointing, Direct Input).- Post-Survey: Quantitative rating of interaction techniques
• User experiment results:- Good acceptance of the prototype in general, but strongly depending on
the used interaction technique
- Initial problems with the workflow of the interaction (Action and Parameter tag configuration, no attention to description)
- Uncertainty about interaction order on the poster
- Many participants wanted to use the interface directly for inputs although having been advised to use Touching or Pointing
Sven Siorpaes, Gregor Broll, 26.07.2006 44/48
User Study and Evaluation II
0
1
2
3
4
5
6
7
8
EasyHandling
Funny Innovative Reliable
Num
ber o
f par
ticip
ants
Applies completelyRather appliesDo not knowRather not appliesNot applies at all
0
1
2
3
4
5
6
7
EasyHandling
Funny Innovative Reliable
Num
ber
of p
artic
ipan
ts
Applies completelyRather appliesDo not knowRather not appliesNot applies at all
0
1
2
3
4
5
6
7
EasyHandling
Funny Innovative Reliable
Num
ber o
f par
ticip
ants
Applies completelyRather appliesDo not knowRather not appliesNot applies at all
Touching Pointing
Direct Input
0 1 2 3 4 5
Touching
Pointing
Direct Input
ReliableInnovativeFunnyEasy Handling
School Marks:
Comparison
Sven Siorpaes, Gregor Broll, 26.07.2006 45/48
Conclusion and Future Work
Sven Siorpaes, Gregor Broll, 26.07.2006 46/48
Conclusion
• Generic approach and architecture for a framework enabling Physical Mobile Interactions with the Internet of Things:
- Support for the complete Physical Mobile Interaction workflow
- Adaptation of SWS to model complex interactions, e.g. interaction sequence
- Flexible abstract type system to link physical objects with service parameters
- Automatic user interface generation from service descriptions and annotations
- Abstract UI widget model independent from specific platforms
• UI adaptation to J2ME runtime and XHTML browser
• J2ME client prototype supporting the interaction techniques Touching, Pointing and Direct Input
Sven Siorpaes, Gregor Broll, 26.07.2006 47/48
Future Work
Framework- Assumtions are made regarding the message format of the service => Flexible interpretation
of Web Services necessary to extend existing Web Services, e.g. Ebay Web Service
- Extend use of context information
- Extend the Abstract Widget Type Model to other widgets and modalities
Mobile Client• Solve issues identified in the second high-fidelity user study
• Implementation of other interaction techniques
Physical Mobile Interaction (Posters)- Cross-poster interactions; parameter-tags could be used with other posters
- General tag could describe poster and its tags => input for generic service search
Authoring- Tedious and error-prone creation of service description and service annotation, only partially
automated
- Desired: Fully automated creation of descriptions => Project Thesis of Christian Braun
Sven Siorpaes, Gregor Broll, 26.07.2006 48/48
Questions?Thank You!
Recommended