View
215
Download
1
Category
Preview:
Citation preview
Information Systems & Semantic Web
University of Koblenz ▪ Landau, Germany
Semantic Multimedia Web
Ansgar Scherp
Basierend auf Folien von Carsten Saathoff, Raphael Troncy und Lynda Hardman
ISWeb - Information Systems & Semantic Web
Carsten Saathoffsaathoff@uni-koblenz.de
MMDB 02.06.08Slide 2
Was bisher geschah...
MMDB als Erweiterung von ORDBMS Information Retrieval als Basis für Queries Feature Extraktion um Inhalt zu beschreiben Feature Transformation um kompaktere Darstellung zu
bekommen Fokus auf low-level features Distanzen und Ähnlichkeiten Indizierung von Features für schnellen Zugriff
ISWeb - Information Systems & Semantic Web
Carsten Saathoffsaathoff@uni-koblenz.de
MMDB 02.06.08Slide 3
Probleme traditioneller MMDB
Datenstrukturen und Schemata meist proprietär MMDB typischerweise für eine Applikation aufgesetzt. Spätere Integration mit anderen Applikationen schwer Ad-Hoc Integration eher unmöglich
Starker Fokus auf Low-Level Features Semantische Lücke
• Kein direktes Mapping zwischen Low-Level Features und Semantik des Bildes
Retrieval primär über Ähnlichkeit• Fast alle Studien zeigen, dass Nutzer dadurch nicht
zufriedengestellt werden• Nutzer wollen semantisch Anfragen
ISWeb - Information Systems & Semantic Web
Carsten Saathoffsaathoff@uni-koblenz.de
MMDB 02.06.08Slide 4
Metadaten
ISWeb - Information Systems & Semantic Web
Carsten Saathoffsaathoff@uni-koblenz.de
MMDB 02.06.08Slide 5
Metadaten (2)
Stichworte
GPS InformationKamera DatenDatum
ISWeb - Information Systems & Semantic Web
Carsten Saathoffsaathoff@uni-koblenz.de
MMDB 02.06.08Slide 6
Metadaten
Daten über Daten Autor, Creation-Date, Keywords, ...
Wie repräsentieren? Relationales Schema XML Semantic Web Technologien
Metadaten sollten interoperabel sein Web Desktop Intranets Viele Applikation müssen kommunizieren
ISWeb - Information Systems & Semantic Web
Carsten Saathoffsaathoff@uni-koblenz.de
MMDB 02.06.08Slide 7
Überblick
Semantic Web + Multimedia Semantische Lücke Canonical Process for Multimedia Production MPEG7 und COMM
Probleme mit MPEG7 Core Ontology on Multimedia (COMM)
Linked Open Data KAT – K-Space Annotation Tool
Semi-Automatische Effiziente Annotation Szenarien
ISWeb - Information Systems & Semantic Web
Carsten Saathoffsaathoff@uni-koblenz.de
MMDB 02.06.08Slide 8
Semantic Web auf einer Folie
EmployeeEmployee
PostDocPostDoc ProfessorProfessor
PersonPerson
rdfs:subClass rdfs:subClass
rdfs:subClass
cooperatesWithcooperatesWith
rdfs:Range rdfs:DomainOntology
<swrc:Professor rdf:ID="person_sst"> <swrc:name>Steffen Staab </swrc:name>...</swrc:Professor>
http://www.uni-koblenz.de/~staab
rdf:typerdf:type
Meta-data
<swrc:PostDoc rdf:ID="person_sha"> <swrc:name>Siegfried Handschuh</swrc:name>
...</swrc:PostDoc>
Web page
http://www.deri.ie/~shaURL
<swrc:cooperatesWith rdf:resource = "http://www.uni-koblenz.de/~staab/#person_sst"/>
swrc:cooperatesWith
ISWeb - Information Systems & Semantic Web
Carsten Saathoffsaathoff@uni-koblenz.de
MMDB 02.06.08Slide 9
Semantic Web for Multimedia
IsWeb @ Bad Kreuznach 2007
WIAMIS 2008 in Klagenfurt
Multimedia Ontology
http://kodemaniak.de/foaf.rdf
depicts
depicts
„Carsten Saathoff“
Domain Ontology
hasName
rdf:type
Researcher
ResearchMeeting:=>=1 depicts.Researcher
ResearchMeeting
Zeig mir alle Bilder von Research Meetings!
http://isweb.uni-koblenz.de/
http://wiamis2008.org
ISWeb - Information Systems & Semantic Web
Carsten Saathoffsaathoff@uni-koblenz.de
MMDB 02.06.08Slide 10
Überblick
Semantic Web + Multimedia Semantische Lücke Canonical Process for Multimedia Production MPEG7 und COMM
Probleme mit MPEG7 Core Ontology on Multimedia (COMM)
Linked Open Data KAT – K-Space Annotation Tool
Semi-Automatische Effiziente Annotation Szenarien
ISWeb - Information Systems & Semantic Web
Carsten Saathoffsaathoff@uni-koblenz.de
MMDB 02.06.08Slide 11
Semantische Lücke
0010EE -> bläulich
0033FE -> bläulich
Visuell ähnlich! Aber semantischunterschiedlich!
ISWeb - Information Systems & Semantic Web
Carsten Saathoffsaathoff@uni-koblenz.de
MMDB 02.06.08Slide 12
Semantische Lücke
Visuell ähnlich, semantisch ähnlich, aber...
Italien USA
ISWeb - Information Systems & Semantic Web
Carsten Saathoffsaathoff@uni-koblenz.de
MMDB 02.06.08Slide 13
Ebenen der Semantik
Generische Objekte
Generische Szene
Spezifische Objekte
Spezifische Szene
Abstrakte Objekte
Abstrakte Szene
Eine Person
Personen unterhaltensich
Churchill
Churchill, Roosevelt, Stalinsitzen zusammen
Churchill, Premierminister, GB, ...
Big Three, Yalta KonferenzWWII, ...
Wis
sen
ISWeb - Information Systems & Semantic Web
Carsten Saathoffsaathoff@uni-koblenz.de
MMDB 02.06.08Slide 14
Überblick
Semantic Web + Multimedia Semantische Lücke Canonical Process for Multimedia Production MPEG7 und COMM
Probleme mit MPEG7 Core Ontology on Multimedia (COMM)
Linked Open Data KAT – K-Space Annotation Tool
Semi-Automatische Effiziente Annotation Szenarien
ISWeb - Information Systems & Semantic Web
Carsten Saathoffsaathoff@uni-koblenz.de
MMDB 02.06.08Slide 15
Overview of Canonical Processes
ISWeb - Information Systems & Semantic Web
Carsten Saathoffsaathoff@uni-koblenz.de
MMDB 02.06.08Slide 23
Example 2: Vox Populi Video Sequences Generation
Stefano Bocconi, Frank Nack
Interview with America video footage with interviews and background material about the opinion of American people after 9-11 http://www.interviewwithamerica.com
Example question:What do you think of the war in Afghanistan?
“I am never a fan of military action, in the big picture I don’t think it is ever a good thing, but I think there are circumstances in which I certainly can’t think of a more effective way to counter this sort of thing…”
ISWeb - Information Systems & Semantic Web
Carsten Saathoffsaathoff@uni-koblenz.de
MMDB 02.06.08Slide 24
Vox Populi Premeditate Process
Analogous to the pre-production process in the film industry Static versus dynamic video artifact
Output Script, planning of the videos to be captured Questions to the interviewee prepared Profiles of the people interviewed:
education, age, gender, race Locations where the interviews take place
Premeditate
ISWeb - Information Systems & Semantic Web
Carsten Saathoffsaathoff@uni-koblenz.de
MMDB 02.06.08Slide 25
Vox Populi Annotations
Contextual Interviewee (social), locations
Descriptive Question asked and transcription of the answers Filmic continuity, examples:
• gaze direction of speaker (left, centre, right)• framing (close-up, medium shot, long shot)
Rhetorical Rhetorical Statement Argumentation model: Toulmin model
ISWeb - Information Systems & Semantic Web
Carsten Saathoffsaathoff@uni-koblenz.de
MMDB 02.06.08Slide 26
Vox Populi Statement Annotations
Statement formally annotated: <subject> <modifier> <predicate> E.g. “war best solution”
A thesaurus containing: Terms on the topics discussed (155) Relations between terms: similar (72), opposite (108),
generalization (10), specialization (10) E.g. war opposite diplomacy
ISWeb - Information Systems & Semantic Web
Carsten Saathoffsaathoff@uni-koblenz.de
MMDB 02.06.08Slide 27
Toulmin Model
ClaimData
Qualifier
Warrant
Backing
Condition
Concession
57 Claims, 16 Data, 4 Concessions, 3 Warrants, 1 Condition
ISWeb - Information Systems & Semantic Web
Carsten Saathoffsaathoff@uni-koblenz.de
MMDB 02.06.08Slide 28
Vox Populi Query Interface
Query
Construct Message
ISWeb - Information Systems & Semantic Web
Carsten Saathoffsaathoff@uni-koblenz.de
MMDB 02.06.08Slide 29
Vox Populi Organize Process
Using the thesaurus, create a graph of related statements nodes are the statements
(corresponding to video segments)“war best solution”,“diplomacy best solution”,“war not solution”
edges are either support or contradict
support
contradict
war best solution
war not solution
diplomacy best solution
Organize
ISWeb - Information Systems & Semantic Web
Carsten Saathoffsaathoff@uni-koblenz.de
MMDB 02.06.08Slide 30
Result of Vox Populi Query
I am not a fan of military actions
War has neversolvedanything
I cannot think of a more effective solution
Two billionsdollar bombson tentsDistribute
Publish
ISWeb - Information Systems & Semantic Web
Carsten Saathoffsaathoff@uni-koblenz.de
MMDB 02.06.08Slide 31
Vox Populi Processes
ISWeb - Information Systems & Semantic Web
Carsten Saathoffsaathoff@uni-koblenz.de
MMDB 02.06.08Slide 32
Canonical Processes 101
Canonical: reduced to the simplest and most significant form possible without loss of generality
Formalization of each process in UML diagrams Process Process artifacts Process actors External world artifacts
ISWeb - Information Systems & Semantic Web
Carsten Saathoffsaathoff@uni-koblenz.de
MMDB 02.06.08Slide 34
Create Media Asset
Process where media assets are captured, generated or transformed
C apturing Devic e<<process actor>>
C apturing<<process>>
1..*1..*
<<involves>>
G eneration Program<<process actor>>
G eneration<<process>>
1..*1..*
<<involves>>
Editing Program<<process actor>>
Transforming<<process>>
1..*1..*
<<involves>>
Premeditate A rtifac ts(from Premedita te )
<<process artifact>>
C reation A c tor<<process actor>>
Message(from Construct M essage)
<<process artifact>>
Media A sset<<media asset>>
C reate Media A sset<<process>>
<<input>>
1..*1..*
<<involves>>
<<input>>
1..*1..*<<output>>
ISWeb - Information Systems & Semantic Web
Carsten Saathoffsaathoff@uni-koblenz.de
MMDB 02.06.08Slide 36
Semantic Annotate
V oc abulary Term<<term>>
Semantic A nnotate<<process>>
V oc abulary<<schema>> <<input>>
Semantic A rtifac t A nnotation<<annotation>>
desc ribed in term s of
<<output>>
link
A ny Proc ess A rtifac t<<process artifact>>
A nnotate<<process>>
**
<<input>>
A rtifac t A nnotation<<annotation>>
1..*1..*
<<output>>
The annotation uses some controlled vocabularies Subject matter annotations of your photos Rhetorical annotations in Vox Populi
ISWeb - Information Systems & Semantic Web
Carsten Saathoffsaathoff@uni-koblenz.de
MMDB 02.06.08Slide 37
Package
Process where process artifacts are logically and physically packed
Physic al Package<<process>>
Logic al Pac kage<<process>>
Input given by a user<<external w orld artifact>>
Package<<process>>
**<<input>>
A ny Process A rtifac t<<process artifact>>
**<<input>>
Multimedia Package<<composite a rtifact>>
1..*1..*
<<output>>
**
contains
ISWeb - Information Systems & Semantic Web
Carsten Saathoffsaathoff@uni-koblenz.de
MMDB 02.06.08Slide 43
Canonical Processes Possible Flow
ISWeb - Information Systems & Semantic Web
Carsten Saathoffsaathoff@uni-koblenz.de
MMDB 02.06.08Slide 44
Sum Up
Community agreement, not “yet another model”Large proportion of the functionality provided by multimedia
applications can be described in terms of this modelInitial step towards the definition of open web-based data
structures for describing and sharing semantically annotated media assets
ISWeb - Information Systems & Semantic Web
Carsten Saathoffsaathoff@uni-koblenz.de
MMDB 02.06.08Slide 45
Überblick
Semantic Web + Multimedia Semantische Lücke Canonical Process for Multimedia Production MPEG7 und COMM
Probleme mit MPEG7 Core Ontology on Multimedia (COMM)
Linked Open Data KAT – K-Space Annotation Tool
Semi-Automatische Effiziente Annotation Szenarien
ISWeb - Information Systems & Semantic Web
Carsten Saathoffsaathoff@uni-koblenz.de
MMDB 02.06.08Slide 46
MPEG7
ISO Standard der MPEG Community Einheitliches Format zu Speicherung von Multimedia
Metadaten Struktur Features Semantik
Extrem (!) umfangreich Darauf basierend MPEG21 mit Fokus auf DRM etc. Hat im Gegensatz zu MPEG1-4 nichts mit Kodierung zu tun
ISWeb - Information Systems & Semantic Web
Carsten Saathoffsaathoff@uni-koblenz.de
MMDB 02.06.08Slide 47
MPEG7 (2)
Basiert auf XML MPEG7 Beschreibung ist eine Hierarchie
Deskriptoren beschreiben Eigenschaften von Multimedia Daten Struktur
• Video -> Shots -> Frames • Bilder -> Segmente
Semantik Low-Level Features
Um Flexibilität zu gewahren, können Deskriptoren sehr vielseitig kombiniert werden.
ISWeb - Information Systems & Semantic Web
Carsten Saathoffsaathoff@uni-koblenz.de
MMDB 02.06.08Slide 48
Big Three
ISWeb - Information Systems & Semantic Web
Carsten Saathoffsaathoff@uni-koblenz.de
MMDB 02.06.08Slide 49
Issues<Mpeg7> <Description xsi:type="ContentEntityType"> <MultimediaContent xsi:type=„ImageType"> <Image> <SpatialDecomposition>
<StillRegion id=„SR1“> <TextAnnotation> <KeywordAnnotation xml:lang="en"> <Keyword>Churchill</Keyword> </KeywordAnnotation> </TextAnnotation></StillRegion>
<StillRegion id=„SR2“> <Semantic> <Label> <Name>Roosevelt</Name> <Label> </Semantic></StillRegion>
<StillRegion id=„SR3“> <Semantic> <Definition> <!-- Also TextAnnotation!! --> <StructuredAnnotation> <WhatObject> <Name xml:lang="en">Stalin</Name> </WhatObject> </StructuredAnnotation> </Definition> </Semantic> </StillRegion>...
How do you formulate a query to get images
showing Churchill et al.?
First Shot (Xpath)://StillRegion[.//Keyword=“Churchill” or
.//Keyword=”Roosevelt” or
.//Keyword=”Stalin”]
Winston ChurchillRecognizer
Franklin RooseveltRecognizer
Josef StalinRecognizer
ISWeb - Information Systems & Semantic Web
Carsten Saathoffsaathoff@uni-koblenz.de
MMDB 02.06.08Slide 50
Probleme mit MPEG-7?
Annotationen sind nicht interoperabel! Mehrdeutigkeiten Mehrere Möglichkeiten um semantisch identische Annotationen
zu beschreiben Deskriptoren können auf viele Arten kombiniert werden
Komplexe Anfragen müssen alle Alternativen beachten
ISWeb - Information Systems & Semantic Web
Carsten Saathoffsaathoff@uni-koblenz.de
MMDB 02.06.08Slide 51
Capabilities and Maturity Levels
Integration Automation
Former Situation Current Situation Future / Desired Situation
no standard, no vocabulary manual 1:1 agreement on
format and semantics tight coupling of data and
applications
standard vocabulary manual 1:1 agreement on
mpeg-7 vocabulary tight coupling of data and
applications
standard vocabulary pre-defined meaning ad-hoc coupling of data
and applications CORE ONTOLOGY
Nächster Teil der VL
Formerly
MPEG-7
COMM
ISWeb - Information Systems & Semantic Web
Carsten Saathoffsaathoff@uni-koblenz.de
MMDB 02.06.08Slide 52
Ontology Stack
Core Ontologies
Domain Ontologies
Foundational Ontologies
Foundational Ontologies Span across multiple fields, each covering multiple domains Modelling of the most abstract concepts like event, object, ...
Core Ontologies Situated in one field, but spans multiple domains Can base on foundational ontologies Examples fields: events, annotation, communication, ...
Domain Ontology For a specific domain, e.g., fishery, human body, etc.
ISWeb - Information Systems & Semantic Web
Carsten Saathoffsaathoff@uni-koblenz.de
MMDB 02.06.08Slide 53
MPEG-7
COMM
Requirements on a high quality
MM OntologyChallenge
BuildingBlock
Legend
ISWeb - Information Systems & Semantic Web
Carsten Saathoffsaathoff@uni-koblenz.de
MMDB 02.06.08Slide 54
TextDescriptor
MusicManager
CompoundDocument
Requirements for COMM
ReusabilityDesign a core ontology for any multimediarelated application
MPEG-7-ComplianceSupport most important description tools
ExtensibilityEnable inclusion of further
• description tools(even those that are not part of MPEG-7!)
• media types Separation of Concerns
Clear separation of domain knowledge andknowledge about structure
ModularityEnable customization of multimedia ontology
High degree of axiomatization Ensure interoperability throughmachine accessible semantics Churchill
RecognizerJosef StalinRecognizer
FaceDetector
PhotoManager
AuthoringTool
SemanticAnnotation
decomposition visual descriptors
audio descriptors ...
<Mpeg7> ...</Mpeg7>
<Mpeg7> ...</Mpeg7>
ISWeb - Information Systems & Semantic Web
Carsten Saathoffsaathoff@uni-koblenz.de
MMDB 02.06.08Slide 55
Is MPEG-7 a good Basis for a high Quality Ontology?
Shortcomings of badly modelled ontologies [Oberle et al., 2006]:1) Conceptual ambiguity
• Difficulties in understanding themeaning of concepts and theirrelations
2) Poor axiomatization• Axiomatization of well defined
concepts is missing3) Loose Design
• Presence of modelling artefacts(concepts without ontological meaning)
Shortcomings mainly hinder Extensibility Interoperability
Especially 1) and 2) are major shortcomings of MPEG-7 1-to-1 translations from MPEG-7 to OWL/RDFS (e.g.
[Hunter, 2003a]) will not result in high quality ontologies!
<StillRegion id=„SR1“> <TextAnnotation> <KeywordAnnotation xml:lang="en"> <Keyword>Churchill</Keyword> </KeywordAnnotation> </TextAnnotation></StillRegion>
<StillRegion id=„SR2“> <Semantic> <Label> <Name>Roosevelt</Name> <Label> </Semantic></StillRegion>
<StillRegion id=„SR3“> <Semantic> <Definition> <!-- Also TextAnnotation!! --> <StructuredAnnotation> <WhatObject> <Name xml:lang="en">Stalin </Name> </WhatObject> </StructuredAnnotation> </Definition> </Semantic> </StillRegion>
ISWeb - Information Systems & Semantic Web
Carsten Saathoffsaathoff@uni-koblenz.de
MMDB 02.06.08Slide 56
MPEG-7
COMM
Requirements on a high quality
MM Ontology
Quality of Ontologies
Challenge
BuildingBlock
Legend
ISWeb - Information Systems & Semantic Web
Carsten Saathoffsaathoff@uni-koblenz.de
MMDB 02.06.08Slide 57
How to Design a High Quality Multimedia Ontology?
Approach from [Oberle, 2005], [Oberle et al., 2006]:Use a well designed foundational ontology as a modelling basis to avoid shortcomings
Foundational ontologies provide Formal precision Domain independence Broad scope
Building upon foundational ontologies prevents easy inclusion of modeling artefacts reduces conceptual ambiguity inherit rich axiomatization
ISWeb - Information Systems & Semantic Web
Carsten Saathoffsaathoff@uni-koblenz.de
MMDB 02.06.08Slide 58
Methodology
MPEG-7
COMM
Requirements: High Quality MM Ontology
Quality of Ontologies
Quality Measures for Ontologies
Reference Ontologie
MPEG-7Compliance
Challenge
BuildingBlock
Legend
ISWeb - Information Systems & Semantic Web
Carsten Saathoffsaathoff@uni-koblenz.de
MMDB 02.06.08Slide 59
Methodology for Design Pattern Definition
Identification of most important MPEG-7 functionalities[Arndt et al., 2007]: Decomposition of multimedia content into segments Annotation of segments with meta data (e.g. visual descriptor,
media information, creation & production, …) General:
• Identify repetitive structures and describe them at an abstract level
• Describe digital data by digital data at an arbitrary level of granularity
Additional patterns are needed for: Complex data types of MPEG-7 Semantic annotation by using domain ontologies
Interface between reusable multimedia core and domain specific knowledge
ISWeb - Information Systems & Semantic Web
Carsten Saathoffsaathoff@uni-koblenz.de
MMDB 02.06.08Slide 60
DOLCE Design Patterns: OIO
Foundational ontology DOLCE+DnS Ultralight Aims at capturing the most essential aspects in the world Defines disjunctive upper classes
Event, Object, Quality and Abstract Follows a pattern-oriented approach for ontology design
2 design patterns (extensions) that are especially important for MPEG-7: Ontology of Information objects (OIO): Formalization of information
exchange Information object represents pure abstract information (message) Relevance for multimedia ontology:
• MPEG-7 describes digital data (multimedia information objects) with digital data (annotation)
• Digital data entities are information objects
ISWeb - Information Systems & Semantic Web
Carsten Saathoffsaathoff@uni-koblenz.de
MMDB 02.06.08Slide 61
DOLCE Design Patterns: D&S
Descriptions & Situations (D&S): Formalization of Context
Relevance for multimedia ontology:
Meaning of digital data depends on context
Digital data entities are connected through computational situations (e.g. input and output data of an algorithm)
Algorithms are descriptions
Annotations and decompositions are situations that satisfy the rules of an algorithm / method
ISWeb - Information Systems & Semantic Web
Carsten Saathoffsaathoff@uni-koblenz.de
MMDB 02.06.08Slide 62
Methodology
MPEG-7
COMM
Requirements: High Quality MM Ontology
Quality of Ontologies
Quality Measures for Ontologies
Reference Ontologie
Identification of repetitive structures
MPEG-7Compliance
Pattern definition through
Specialization
Challenge
BuildingBlock
Legend
Repr. of Context
Repr. of Information
ISWeb - Information Systems & Semantic Web
Carsten Saathoffsaathoff@uni-koblenz.de
MMDB 02.06.08Slide 63
Ontology of Information Objects (OIO)
social-object information-encoding-system
information-object description
situationagent particular information-realization
ordered-by
realized-by
interpretedBy
about
setting
satisfiesexpressedBy
Ontology of Information Objects (OIO)
ISWeb - Information Systems & Semantic Web
Carsten Saathoffsaathoff@uni-koblenz.de
MMDB 02.06.08Slide 64
Example
Information Object „Graz Tourist Guide“Information Realization
http://cms.graztourismus.at/cms/ziel/42425/EN/
BookletInformation Encoding:
English German
About: Places, Buildings (e.g. Clock Tower)Agent: 1. iMedia Visitor / 2. Tourist Officer / 3. Graphics DesignerExpresses:
1. Walking Path through Graz2. Small-Size Tourist Guide3. Arrangement of Illustrations
ISWeb - Information Systems & Semantic Web
Carsten Saathoffsaathoff@uni-koblenz.de
MMDB 02.06.08Slide 65
Descriptions & Situations (D&S)
social-objectconcept
parameter
role
course
description
situation
information-objectregion
endurant
perdurant
sequences
valued-bydefines
played-by setting
satisfies
expressed-by
Descriptions & Situations (D&S)
requires
methodsocial-objectconcept
parameter
role
course
description
situation
information-objectregion
endurant
perdurant
sequences
valued-bydefines
played-by setting
satisfies
expressed-by
Descriptions & Situations (D&S)
requires
method
Distinction between: DOLCE ground entities (regions, endurants, perdurants) Descriptive entities (parameters, roles, courses)
Descriptions Formalize context Define descriptive concepts
Situations Are explained by descriptions Are settings for ground entities
ISWeb - Information Systems & Semantic Web
Carsten Saathoffsaathoff@uni-koblenz.de
MMDB 02.06.08Slide 66
Putting it Together: Decomposition Pattern
digital-data
multimedia-data
output-segment-roleplays
processing-role
input-roleoutput-role
segment-decompositionalgorithm
segmentation-algorithm
setting
satisfies
situationmethod
input-segment-role
D&S / OIOdefines
mask-rolerequires
description
structured-data-description
localization-descriptor
plays
information-object role
ISWeb - Information Systems & Semantic Web
Carsten Saathoffsaathoff@uni-koblenz.de
MMDB 02.06.08Slide 67
Example
Image1 playsRole SegmentationInput
Segment2 playsRole SegmOutp
Segment4 playsRole SegmOutp
Segment1 playsRole SegmOutp
Segment3 playsRole SegmOutpSegment1 playsRole SegmInput
Via its role in a computational task the different parts may be arbitrarily nested and related to different computing algorithms
Querying for all subparts takes place along a well-defined pattern
ISWeb - Information Systems & Semantic Web
Carsten Saathoffsaathoff@uni-koblenz.de
MMDB 02.06.08Slide 68
Modular Architecture
Multimedia ontology consists ofCore module that contains the
design patternsModules that specialize the core
module for different media types
Modules that contain media independent MPEG-7 description tools such as media information or creation & production
Data type module that formalizes MPEG-7 data types e.g. matrices, vectors, unsigned-int-5, float-vector, probability-vector, …
DOLCE
Descriptions & Situation
Ontology of Information
Objects
Core
Visual Audio
Datatype
Media
Text / LingInfo
Domain Ontolog
Connected by SemanticAnnotation Pattern
Localization
Multimedia Knowledge (COMM)
Fundamental Knowledge about the World
Knowledge about a specific Domain
ISWeb - Information Systems & Semantic Web
Carsten Saathoffsaathoff@uni-koblenz.de
MMDB 02.06.08Slide 69
Does the Multimedia Ontology fulfil the Requirements?
Reusability MPEG-7-Compliance
Design patterns enable therepresentation of description tools
Extensibility Design patterns are media
independent possibility to include• further media types• arbitrary descriptors
Extensions of multimedia ontologywill not affect legacy annotations dueto DOLCE+D&S+OIO
Separation of Concerns Clear separation between domain
specific and multimedia relatedknowledge
Modularity Modular architecture allows customization
High degree of axiomatization Design patterns come with generic
axiomatization that is refined in derivedontology modules
ChurchillRecognizer
Josef StalinRecognizer
PhotoManager
COMM
One such extension has already beendone for Text Annotation.
Another one for compund documentannotation is currently developed!
Content & Media Annotation Pattern
Semantic Annotation Pattern
See slide before this slide!
OWL-DL version available for download.
ISWeb - Information Systems & Semantic Web
Carsten Saathoffsaathoff@uni-koblenz.de
MMDB 02.06.08Slide 70
Benefits of a DOLCE-aligned Multimedia Ontology
Usage of DOLCE enforces clean design Constraints prohibit arbitrary placement of MPEG-7 concepts into
DOLCE• Similar concepts will be placed on similar locations of the
taxonomy• Things that are different, have to be separated
(e.g. data and the perceivable content that is carried)
Extensibility due to underlying general taxonomy of DOLCEPossibility to describe multimedia domain at an
arbitrary level of detail (e.g. segments have pixels as atomic parts)
Rigorous application of the D&S and OIO patterns allows description of digital data in different contexts(e.g. data acting as input or output for an algorithm)
ISWeb - Information Systems & Semantic Web
Carsten Saathoffsaathoff@uni-koblenz.de
MMDB 02.06.08Slide 71
Benefits compared to MPEG-7
Linkage with domain ontologies allows meaningful semantic annotation of multimedia content Semantic part can be entirely replaced with a domain ontology Clear separation between domain ontologies and multimedia core
ontology through semantic annotation pattern
Easier queries Annotation pattern guarantees equal representation of all
annotations Complex data type pattern guarantees uniform access to nested data
No complex XML-structures to parse Multimedia ontology only uses restricted inventory of DOLCE
predicates
Higher interoperability through machine accessible semantics and underlying DOLCE axiomatization
ISWeb - Information Systems & Semantic Web
Carsten Saathoffsaathoff@uni-koblenz.de
MMDB 02.06.08Slide 72
http://en.wikipedia.org/wiki/Yalta_Conference
World War II
Yalta ...
...History Ontology
“Creating a Multimedia Presentation” Revisited
SR1 SR2 SR3
Winston ChurchillRecognizer
Franklin D. RooseveltRecognizer
Josef StalinRecognizer
ChurchillRooseveltStalin
Sparql: select ?image where { ?image plays AnnotatedDataRole. ?x plays SemanticLabelRole. ?x rdf:type pol:President }
PhotoManager
AuthoringTool
ISWeb - Information Systems & Semantic Web
Carsten Saathoffsaathoff@uni-koblenz.de
MMDB 02.06.08Slide 73
Überblick
Semantic Web + Multimedia Semantische Lücke Canonical Process for Multimedia Production MPEG7 und COMM
Probleme mit MPEG7 Core Ontology on Multimedia (COMM)
Linked Open Data KAT – K-Space Annotation Tool
Semi-Automatische Effiziente Annotation Szenarien
ISWeb - Information Systems & Semantic Web
Carsten Saathoffsaathoff@uni-koblenz.de
MMDB 02.06.08Slide 74
A Giant Graph Open to the World
Annotate the content (interpretation)Elephant, Ganesh, Thailande, Holidays, Chiang Mai
Link to knowledge on the Web:img foaf:depicts dbpedia:Ganeshdbpedia:Ganesh rdfs:label "Vinayaka"dbpedia:Ganesh skos:altlabel "Ganapati" dbpedia:Ganesh rdf:type wn:synset-Deities-noun-1dbpedia:Ganesh owl:sameas wn:synset-Ganesh-noun-1
<rdf:Description rdf:about="Ganesh.jpg"> <dc:title>An image of the Elephant Ganesh</dc:title> <dc:creator>Raphaël Troncy</dc:creator> </rdf:Description>
ISWeb - Information Systems & Semantic Web
Carsten Saathoffsaathoff@uni-koblenz.de
MMDB 02.06.08Slide 75
Linking Open Data Project
Expose open datasets in RDF
Set RDF links among the data items for different datasets
Over 2 billion triples, 3 millions links (March 2008)
http://richard.cyganiak.de/2007/10/lod/
ISWeb - Information Systems & Semantic Web
Carsten Saathoffsaathoff@uni-koblenz.de
MMDB 02.06.08Slide 76
Linked Open Data March 2009
ISWeb - Information Systems & Semantic Web
Carsten Saathoffsaathoff@uni-koblenz.de
MMDB 02.06.08Slide 77
Warum ist LOD wichtig?
• RDFa wird von der Google Suchmaschine verarbeitet• Content provider bieten nun RDFa an• Erhöht Click-trough-rate auf Webseiten (Werbeanzeigen)• Erhöht Ranking der Webseiten in Google
• SIOC Ontology zur Verlinkung von Online Communities wird genutzt von Yahoo! • Wird durch SearchMonkey eingesammelt• Tools um RDFa zu publishen
• Usw.
ISWeb - Information Systems & Semantic Web
Carsten Saathoffsaathoff@uni-koblenz.de
MMDB 02.06.08Slide 78
DBpedia
DBpedia is a community effort to: extract structured "infobox" information from Wikipedia interlink DBpedia with other datasets on the Web
ISWeb - Information Systems & Semantic Web
Carsten Saathoffsaathoff@uni-koblenz.de
MMDB 02.06.08Slide 79
DBpedia
ISWeb - Information Systems & Semantic Web
Carsten Saathoffsaathoff@uni-koblenz.de
MMDB 02.06.08Slide 80
Automatic Links Among Open Datasets
Processors can switch automatically from one to the other …
ISWeb - Information Systems & Semantic Web
Carsten Saathoffsaathoff@uni-koblenz.de
MMDB 02.06.08Slide 81
Take Home Message
Reuse what is there Of course, one could create RDF data manually …
… but that is unrealistic on a large scale Goal is to generate RDF data automatically when
possible and "fill in" by hand only when necessary• service to get RDF from flickr images
http://www.kanzaki.com/works/2005/imgdsc/flickr2rdf• service to get RDF from XMP
http://www.ivan-herman.net/cgi-bin/blosxom.cgi/WorkRelated/SemanticWeb/xmpextract.html
Expose what you make
ISWeb - Information Systems & Semantic Web
Carsten Saathoffsaathoff@uni-koblenz.de
MMDB 02.06.08Slide 82
Überblick
Semantic Web + Multimedia Semantische Lücke Canonical Process for Multimedia Production MPEG7 und COMM
Probleme mit MPEG7 Core Ontology on Multimedia (COMM)
Linked Open Data KAT – K-Space Annotation Tool
Semi-Automatische Effiziente Annotation Szenarien
ISWeb - Information Systems & Semantic Web
Carsten Saathoffsaathoff@uni-koblenz.de
MMDB 02.06.08Slide 83
KAT: K-Space Annotation Tool
Goal Efficient annotation of multimedia content Means to create semantically rich annotations
KAT provides framework for Executing analysis plugins Providing visualisation plugins
• Displaying/annotating content• Browsing
Interfaces with Core Ontology for Multimedia (COMM)• Provides the common model
Role based messaging to leverage reuse of components
ISWeb - Information Systems & Semantic Web
Carsten Saathoffsaathoff@uni-koblenz.de
MMDB 02.06.08Slide 84
Efficient Annotation
Reduce time required by user for annotating content
Integration of Automatic analysis methods
• Region labeling, object detection• Key Frame Extraction, Shot Boundary Detection
Automatic Organisation• Clustering
Inferencing• Based on formal domain ontologies
Semi-automation
ISWeb - Information Systems & Semantic Web
Carsten Saathoffsaathoff@uni-koblenz.de
MMDB 02.06.08Slide 85
Semantically Rich Annotations
Relational Annotation Express how depicted entities are related Example: Soccer Game
• Who is tackling whom?• Why was the penalty given?
Ontologies provide means to express relations KAT aims at providing the means to efficiently create them
Event-Based annotation Events are prominent in multimedia Create and manage events Relate events and media Allow for event-based retrieval and exploration
ISWeb - Information Systems & Semantic Web
Carsten Saathoffsaathoff@uni-koblenz.de
MMDB 02.06.08Slide 86
Architecture
KAT-Core
Plugin Plugin
GUI
Plugin Viewregister
COMM
Repository Repository...
...
store
display
ISWeb - Information Systems & Semantic Web
Carsten Saathoffsaathoff@uni-koblenz.de
MMDB 02.06.08Slide 87
Szenarien
Effiziente Annotation von persönlichen Bildern Semi-Automatische, Semantische Annotation von Sport-
Ereignissen Browsing von Bildsammlungen im Web
ISWeb - Information Systems & Semantic Web
Carsten Saathoffsaathoff@uni-koblenz.de
MMDB 02.06.08Slide 88
Persönliche Bildsammlungen
Heutzutage typischerweise in Ordnern auf Festplatte Wenig Annotationen weil zu aufwendig Annotationen enthalten
Viel Hintergrund Wissen Gefühle und persönliche Momente
Frage: Wie kann ein Nutzer hier unterstützt werden? Automatische Annotation liefert nur einfache Semantik Daher: Clustering um Ereignisse zu finden User kann dann ganze Ereignisse annotieren Verwendung von NLP (Textanalyse) um semantische
Annotation zu erzeugen
ISWeb - Information Systems & Semantic Web
Carsten Saathoffsaathoff@uni-koblenz.de
MMDB 02.06.08Slide 89
Sportereignisse
Kombination verschiedener Algorithmen Highlight Detection: Goals, Corner-Shot, ...
• Features: Motion, Geräusche, sichtbare Konzepte Analyse von Minute-by-Minute Reports
• Liefert andere Ereignisse Ergebnisse oft nur global
Manuelles Refinement Zuweisen von Namen zu Spielern im Video/Bild Zuweisen von Rollen:
• Wer hat das Faul begangen, wer war Opfer Verknüpfen von Ereignissen
Ziel: möglichst vollständige Annotationen effizient erstellen
ISWeb - Information Systems & Semantic Web
Carsten Saathoffsaathoff@uni-koblenz.de
MMDB 02.06.08Slide 90
Browsen von Bildkollektionen
Flickr als COMM Maping von Tags in Wordnet
Wordnet: linguistische Ontologie Mapping von Geoinformationen nach Geonames
Ontologie von geographischen Informationen (Länder, Orte, ...)
Mapping nach dbpedia Wikipedia als maschienenlesbare Version
Tags bekommen Kontext Anzeige werwandter Bilder, komplexe Queries Ergänzen der Annotationen
ISWeb - Information Systems & Semantic Web
Carsten Saathoffsaathoff@uni-koblenz.de
MMDB 02.06.08Slide 91
91
KAT as basis for der User InterfaceObjectives
Explore and visualize semantic Web 2.0 data in real-time Acquainting oneself about an area of interest
Semantic data 1 billion triples from DBpedia, GeoNames, WordNet,
FOAF files and Flickr Very large, mixed-quality, semantically heterogeneous
Winner of Billion Triples Track, Semantic Web Conference 2008, Karlsruhe [ISWC2008]
ISWeb - Information Systems & Semantic Web
Carsten Saathoffsaathoff@uni-koblenz.de
MMDB 02.06.08Slide 92
92
SemaPlorer – Web 2.0 Content Brower
Search for
locations, persons and tags
Active facets like tags, location
Map showing locations, sights, pictures
Geo-referenced image from Flickr
Information on locations, persons, tags
SemaPlorer Live!http://btc.isweb.uni-koblenz.de/
ISWeb - Information Systems & Semantic Web
Carsten Saathoffsaathoff@uni-koblenz.de
MMDB 02.06.08Slide 93
Werbeblock
• HiWi-Jobs für SemaPlorer++• Interesse an Arbeit in einer Gruppe• Tätigkeit die Kenntnisse aus dem Studium (und darüber
hinaus) praktischen anwenden lässt• Spaß am “Tütfteln” • Entwicklung mit Java
• Mail mit Beschreibung an Erfahrungen / Lebenslauf an scherp@uni-koblenz.de
ISWeb - Information Systems & Semantic Web
Carsten Saathoffsaathoff@uni-koblenz.de
MMDB 02.06.08Slide 94
Werbeblock
• ImageAtlas II Projektpraktikum• Entwicklung einer Plattform zum Taggen und zur Diskussion
von Bildern der Kunstgeschichte und Bildwissenschaft• Zusammen mit dem Institut für Kunstwissenschaft
• Anmeldung: Jetzt über KLIPS• http://isweb.uni-koblenz.de/ -> Lehre -> WS09/10• Fragen? Mail an scherp@uni-koblenz.de
ISWeb - Information Systems & Semantic Web
Carsten Saathoffsaathoff@uni-koblenz.de
MMDB 02.06.08Slide 95
Werbeblock
• Diplomarbeiten• Beispiele für Themen
• Kontextsensitive Visualisierung von Events und Objekten auf der Karte
• Repräsentation von dynamischen organisationalen Prozessen am Beispiel des Notfallmanagements
• TripleRanked Faceted Browsing Interface of Linked Open Data
• Auch online unterhttp://isweb.uni-koblenz.de/interactiveweb
• Und weitere Themen …
Recommended