View
268
Download
2
Embed Size (px)
DESCRIPTION
Presentation of the DM2E Interoperability platform – Omnom & Pubby at the ExLibris Semantic Web Special Interest Group (18 March 2014)
Citation preview
2014-03-18 Konstantin Baierer: Omnom & Pubby 1/36
DM2E Interoperability
platform–
Omnom & Pubby
Konstantin Baierer
2014-03-18 Konstantin Baierer: Omnom & Pubby 2/36
● Components of the DM2E Software Stack● What is Omnom?
– Modelling workflows
– HTTP Protocol
– Provenance
● What is Pubby● Example Pubby setup● Omnom/Pubby Demo
Agenda
2014-03-18 Konstantin Baierer: Omnom & Pubby 3/36
Omnom and Pubby in DM2E
2014-03-18 Konstantin Baierer: Omnom & Pubby 4/36
Omnom and Pubby in DM2E
Omnom
2014-03-18 Konstantin Baierer: Omnom & Pubby 5/36
Omnom and Pubby in DM2E
PubbyOmnom
2014-03-18 Konstantin Baierer: Omnom & Pubby 6/36
● dm2e-ws– Workflow Engine
– Web Services
● dm2e-gui– Graphical User Interface (Omnom / dm2e-gui)
● dm2e-silk– Linking
● dm2e-pubby– RDF Browser
Components of WP2 Interop Platform
● dm2e-ontologies– Data models/ Validation
● dm2e-direct– Command line ingestion
● dm2e-josso– Single Sign On
2014-03-18 Konstantin Baierer: Omnom & Pubby 7/36
● dm2e-ws– Workflow Engine
– Web Services
● dm2e-gui– Graphical User Interface (Omnom / dm2e-gui)
● dm2e-silk– Linking
● dm2e-pubby– RDF Browser
Components of WP2 Interop Platform
Omn m
● dm2e-ontologies– Data models/ Validation
● dm2e-direct– Command line ingestion
● dm2e-josso– Single Sign On
2014-03-18 Konstantin Baierer: Omnom & Pubby 8/36
● dm2e-ws– Workflow Engine
– Web Services
● dm2e-gui– Graphical User Interface (Omnom / dm2e-gui)
● dm2e-silk– Linking
● dm2e-pubby– RDF Browser
Components of WP2 Interop Platform
Omn m
● dm2e-ontologies– Data models/ Validation
● dm2e-direct– Command line ingestion
● dm2e-josso– Single Sign On
2014-03-18 Konstantin Baierer: Omnom & Pubby 9/36
Omnom: Workflows and Configurations
● Workflow– Abstract description of algorithm
– „Blueprint“ or „Data Wiring“
– Less than a dozen for DM2E
● Configuration– Instantiation of a workflow with concrete values
– One for every ingestion
2014-03-18 Konstantin Baierer: Omnom & Pubby 10/36
Omnom: What's in a Workflow?
● Typed Input and Output Parameters● Positions wrapping Web Services
Input and Output Parameters
Iterative positions for repeatable output● Connectors wiring the parameters
=> Define data flow
2014-03-18 Konstantin Baierer: Omnom & Pubby 11/36
Simple XSLT Publish Workflow→
2014-03-18 Konstantin Baierer: Omnom & Pubby 12/36
Simple XSLT Publish Workflow→
Workflow Input Workflow Output
2014-03-18 Konstantin Baierer: Omnom & Pubby 13/36
Simple XSLT Publish Workflow→
Workflow Input Workflow Output
Input
Web Service
Output
Position 1
Input Web Service
OutputPosition 2
2014-03-18 Konstantin Baierer: Omnom & Pubby 14/36
Simple XSLT Publish Workflow→
Workflow Input Workflow Output
Input
Web Service
Output
Position 1
Input Web Service
OutputPosition 2
Connectors
2014-03-18 Konstantin Baierer: Omnom & Pubby 15/36
Omnom: What's in a configuration?
● Parameter Values– Resolvable URLs
– Strings
● Examples– URL of result RDF/XML
– URL of input XSLT
– Name of a dataset for URI generation
– Name-Value pair for XSLT parameters
2014-03-18 Konstantin Baierer: Omnom & Pubby 16/36
Web services: Protocol
● WS describe themselves in RDF:– Input / Output parameters
– Human readable description / label
● WS expect configuration in RDF:– Concrete values for input parameters
– Either resolvable URLs or Literals
● WS immediately return a Job in RDF:– Asynchronously work in the background
– Write (intermediate) results to the Job resource
2014-03-18 Konstantin Baierer: Omnom & Pubby 17/36
Webservice: Execution
2014-03-18 Konstantin Baierer: Omnom & Pubby 18/36
Convention for description/thing Accept conflicts for RDF Inf. Res.
● RDF/XML at http://mydomain/file/123– Document: Output of a transformation
– Metadata: File Size, Creator, MD5 ...
● HTTP Accept: application/rdf+xml– Return the document ?!
– Return the metadata ?!
● Our solution:– No Accept / application/octet-stream => Data
– Accept: RDF/JSON => Metadata
2014-03-18 Konstantin Baierer: Omnom & Pubby 19/36
Types of Services
● Backend Services– Workflow Service
– Configuration Service
– Job Service
– File Services● DM2E File Service● MINT
● Web Services to be used in workflows
2014-03-18 Konstantin Baierer: Omnom & Pubby 20/36
Web Services for Workflows
● Transformation– Transform XML with XSLT to RDF/XML
– Transform RDF with SPARQL to RDF
● Publishing– With provenance to Triplestore
● Iteration– Feed every contained resource to another web service
– ZIP of XML files, OAI-PMH harvesting
● Execute a workflow → Nested Workflows
2014-03-18 Konstantin Baierer: Omnom & Pubby 21/36
More complex: ZIP XSLT-ZIP Publish→ →
2014-03-18 Konstantin Baierer: Omnom & Pubby 22/36
Even more complex: Nested Workflows
2014-03-18 Konstantin Baierer: Omnom & Pubby 23/36
Provenance in DM2E
● All Omnom resources are described in RDF● All results are accesible on the Web
– Either Directly in RDF
– Or as Web Documents, with RDF metadata
● All data is in versioned datasets● [ Dm2e-pubby offers version-aware API ] ● Consumers should "Follow their nose"
2014-03-18 Konstantin Baierer: Omnom & Pubby 24/36
Provenance: PROV Omnom↔
2014-03-18 Konstantin Baierer: Omnom & Pubby 25/36
Omnom Web User Interface
● Single-Page Application● BackboneJS
– MVC framework
– Client/Server: Same data representation: JSON-LD (ish, kind of)
● Bootstrap– Looks like Twitter
– Sane Defaults
2014-03-18 Konstantin Baierer: Omnom & Pubby 26/36
Pubby
● Frontend for Linked Data● Make triplestores web-accesible
– Dereferenceable URI
– 303 Redirects
– Content-Negotiation
● HTML interface
2014-03-18 Konstantin Baierer: Omnom & Pubby 27/36
DM2E Extensions to Pubby
● CONSTRUCT instead of DESCRIBE– Support Named Graphs
– Configuration defines the relevant subgraph that describes a URI
– Multiple Resources per output graph
● More flexible URIs– 'data' and 'page' configurable
– Dataset-specific URIs
– Regex Matching in URIs
● Generate thumbnails
2014-03-18 Konstantin Baierer: Omnom & Pubby 28/36
Example Pubby Use Case
● Want to expose library data– Book data from triplestore 1
– Subject data from triplestore 2
● Using the same consistent interface● Full example here
http://data.dm2e.eu/pubby-example
2014-03-18 Konstantin Baierer: Omnom & Pubby 29/36
Omnom: Workflow Edit
2014-03-18 Konstantin Baierer: Omnom & Pubby 30/36
Configuration Edit
2014-03-18 Konstantin Baierer: Omnom & Pubby 31/36
Job Page
2014-03-18 Konstantin Baierer: Omnom & Pubby 32/36
Pubby View of the result
2014-03-18 Konstantin Baierer: Omnom & Pubby 33/36
Provenance: Follow your nose
● Resource → Versioned Dataset● Versioned Dataset → Dataset● Versioned Dataset → Publish Job● Publish Job → Publish Job Configuration● Publish Job Configuration → Workflow Job● Workflow Job → Workflow● Workflow → User● …
Let's start here
2014-03-18 Konstantin Baierer: Omnom & Pubby 34/36
Omnom: Next steps
● Performance● UI/UX improvements● Easier deployment
2014-03-18 Konstantin Baierer: Omnom & Pubby 35/36
Pubby: Next steps
● Handle weird URIs more gracefully● Properly handle blank nodes
– Collection Ontology Lists
● Caching (hard)● Re-Merge with upstream Pubby
2014-03-18 Konstantin Baierer: Omnom & Pubby 36/36
Thank You!
● Demos:– http://omnom.dm2e.eu
– http://data.dm2e.eu/data/
● Source Code / Issue Tracker– http://github.com/DM2E/dm2e-ws
– http://github.com/DM2E/dm2e-gui
– http://github.com/DM2E/pubby
● Get in touch:– [email protected]
– http://dm2e.eu, @DM2Europeana