LinkedUp Vidi Competition Presentations and Awards 2014, ESWC 2014

Semantic Analysis of Video Data to Enable Exploratory Search

Dr. Harald SackHasso-Plattner-Institut for IT-Systems Engineering

University of Potsdam

Crete, 25/06/2014

Montag, 26. Mai 14

Harald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, LDW 2011, Magdeburg, 30. Sep. 2011

•How to Search Video Web vs. Archive

•How to Open Up Video Data?Video Analysis Technologies

•How to „Understand“ Video Metadata? Context-Driven Semantic Analysis

•How to Use of Semantic Metadata?Exploratory Search and Intelligent Recommendations

Semantic Analysis of Video Data to Enable Exploratory Search

Montag, 26. Mai 14

Vorlesung Semantic Web, Dr. Harald Sack, Hasso-Plattner-Institut, Universität Potsdam

3

Searching the WebMontag, 26. Mai 14

Vorlesung Semantic Web, Dr. Harald Sack, Hasso-Plattner-Institut, Universität Potsdam

4

Searching the WebMontag, 26. Mai 14

Vorlesung Semantic Web, Dr. Harald Sack, Hasso-Plattner-Institut, Universität Potsdam

5

Montag, 26. Mai 14

Vorlesung Semantic Web, Dr. Harald Sack, Hasso-Plattner-Institut, Universität Potsdam

6

Google Knowledge Graph

= “search results with semantic- search information gathered from a wide variety of sources“

Montag, 26. Mai 14

Harald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, Workshop ,Corporate Semantic Web‘, XInnovations 2011, Berlin, 19. Sep. 2011Google Multimedia Search

Montag, 26. Mai 14

Harald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, LDW 2011, Magdeburg, 30. Sep. 2011

‣Google Multimedia Search relies on text-based metadata and link context

How does Google find Multimedia?

Montag, 26. Mai 14

Harald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, LDW 2011, Magdeburg, 30. Sep. 2011

Seach by Media Content

Montag, 26. Mai 14

Harald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, LDW 2011, Magdeburg, 30. Sep. 2011

The Ordinary Archive is a Small World...

Neil Armstrong

Montag, 26. Mai 14

Harald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, LDW 2011, Magdeburg, 30. Sep. 2011

But, wouldn‘t it be nice, if.....

Neil Armstrong

...but maybe you are also interested in

- Buzz Aldrin (1 videos)

- John Glen (1 video)

- Juri Gagarin (2 videos)

- Richard Nixon (3 videos)

- Apollo 11 (1 video)

- NASA (20 videos)

- Moon (14 videos)

- space exploration (34 videos)

- technology (1.205 videos)

Sorry, no results found for ‘Neil Armstrong‘...

Montag, 26. Mai 14

Harald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, LDW 2011, Magdeburg, 30. Sep. 2011

How to Search in Video Archives?

Montag, 26. Mai 14

vfm - Seminar: Metadatenmanagement in Medienunternehmen, 05. September 2012, Bonn Jörg Waitelonis, Hasso-Plattner-Institut Potsdam

Content-Based Search in Video Archives relies on text-based Metadata

Current Solution: Manual Annotation

Montag, 26. Mai 14

Harald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, LDW 2011, Magdeburg, 30. Sep. 2011 image

VisualConceptDetection

Text Recognition

Visual Analysis

(Selected) Automated Multimedia Analysis

Face Detection

Face Detection

Logo Detection

audio-visual

text / images

Audio-Mining

structuralanalysis

AutomatedSpeech

Recognitionaudio event detection

audio

Montag, 26. Mai 14

Annotation of Audiovisual Data

Metadata Extraction

Metadata (e.g. MPEG-7) ... <SpatialDecomposition> <TextAnnotation> <KeywordAnnotation> <Keyword>Astronaut</Keyword> </KeywordAnnotation> </TextAnnotation> <SpatialMask> <SubRegion> <Polygon> <Coords> 480 150 620 480 </Coords> </Polygon> </SubRegion> </SpatialMask> ... </SpatialDecomposition> ...

• Video data with spatiotemporal Annotations

Neil Armstrong

Montag, 26. Mai 14

Vorlesung Semantic Web, Dr. Harald Sack, Hasso-Plattner-Institut, Universität Potsdam

www.yovisto.com

Montag, 26. Mai 14

Harald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, LDW 2011, Magdeburg, 30. Sep. 2011

• Authoritative Metadata• structured data• semi-structured data

• natural language text • Non-authoritative Metadata

• (free) user tags and comments• restricted vocabularies

• (Media) Analysis Metadata• low level features• high level features

• etc.

How to Determine the Meaning of Metadata?

SemanticAnalysis

reliability

context

pragmatics

location dependency

accuracy

timedependency

level ofabstraction

Montag, 26. Mai 14

Neil Armstrong

Astronaut

is a

Person

is a

Science Occupation

subClassOf

Employment

subClassOf

Entities

Ontologies

has an

,Neil Armstrong‘ is more than just a character string

Kosmonautsame as

Juri Gagarin

is a

is NOT a

!

Montag, 26. Mai 14

Astronaut Person

Neil Armstrong

Science Occupation

Employment

is a is a

is a

is a has a

Web of Data

Montag, 26. Mai 14

Web of Data = Linked Open DataBut what, if there is no trivial unique identification?

Armstronguser tag

Montag, 26. Mai 14

Semantic Web Technologies , Dr. Harald Sack, Hasso Plattner Institute, University of Potsdam

Armstrong

Montag, 26. Mai 14

Semantic Web Technologies , Dr. Harald Sack, Hasso Plattner Institute, University of Potsdam

ArmstrongArmstrong+Moon

Montag, 26. Mai 14

Web of Data = Linked Open DataUnderstanding requires Context

Armstrong

Moon

EagleSpace

Montag, 26. Mai 14

4242 42 4224424242 42 4242Semantic AnalysisSemantics is determined by Context

Context Item

N.Steinmetz, H.Sack: Semantic Multimedia Information Retrieval Based on Contextual Descriptions, 2013

„Armstrong landed the Eagle on the Moon.“Text

SEMEX Multimedia Context Model

Context Dimensions

TemporalContext

SpatialContext

ProvenanceContext

Relevance

determines

Ambiguity

influences

Accuracy

influences

Contextual Description

ClassDiversity

Level of Structure

SourceReliability

SourceDiversity

Montag, 26. Mai 14

Armstrong

George Armstrong Custer

Neil Armstrong

The Armstrong Twins

Armstrong, Florida

Armstrong, Ontario

Armstrong Automobile

Joe Armstrong

Armstrong County, Texass

Armstrong Gun

Craig Armstrong

Armstrong (Moon Crater)

Louis Armstrong

Armstrong Tunnel

Louis Armstrong International Airport

Armstrong‘s Theorem

Sir Thomas Armstrong

Ian Armstrong

Eagle Moon

Eagle (Bird)

Eagle (heraldry)

USCGC Eagle

The Eagle (2011 film)

Eagle (song)

John H. EagleEagle (typeface)

Eagle Falls (Washington)

Eagle (Moon Crater)

Eagle (comic)

Eagle (lunar module)

Eagle TV

Armstrong Tunnel

The Eagle (Pub)

War Eagle

The Eagle (newspaper)

Eagle (racehorse)

Angela EagleLinda Eagle

James Philipp Eagle

95 entities448 entities

Armstrong (British Columbia)Karen Armstrong

Curtis Armstrong

Gillian Armstrong Hilary Armstrong

William L. Armstrong

156 entities

Man on the Moon (film)

Moon (song)

Moon Son-Ri

C Moon

The Moon (Tarot card)

Edgar Moon

Moon OS

Moon (Band)

Moon

Moon 44

Man on the Moon (soundtrack)

William Moon

Lottie Moon

Mr. Moon (song)

Man on the Moon (musical)

Darvin Moon

Moon 83

Francis MoonGary Moon

Robert Charles Moon

Black Moon

Allan Moon

Ban-Ki Moon

Fly me to the Moon (song)

Semantic AnalysisNamed Entity Mapping

„Armstrong landed the Eagle on the Moon.“

Consider all entities within the same context

Montag, 26. Mai 14

Armstrong

George Armstrong Custer

The Armstrong Twins

Armstrong, Florida

Armstrong, Ontario

Armstrong Automobile

Joe Armstrong

Armstrong County, Texass

Armstrong Gun

Craig Armstrong

Armstrong (Moon Crater)

Armstrong Tunnel

Louis Armstrong International Airport

Armstrong‘s Theorem

Sir Thomas Armstrong

Ian Armstrong

Eagle Moon

Eagle (Bird)

Eagle (heraldry)

USCGC Eagle

The Eagle (2011 film)

Eagle (song)

John H. EagleEagle (typeface)

Eagle Falls (Washington)

Eagle (Moon Crater)

Eagle (comic)

Eagle TV

Armstrong Tunnel

The Eagle (Pub)

War Eagle

The Eagle (newspaper)

Eagle (racehorse)

Angela EagleLinda Eagle

James Philipp Eagle

95 entities448 entities

Armstrong (British Columbia)Karen Armstrong

Curtis Armstrong

Gillian Armstrong Hilary Armstrong

William L. Armstrong

156 entities

Man on the Moon (film)

Moon (song)

Moon Son-Ri

C Moon

The Moon (Tarot card)

Edgar Moon

Moon OS

Moon (Band)

Moon 44

Man on the Moon (soundtrack)

William Moon

Lottie Moon

Mr. Moon (song)

Man on the Moon (musical)

Darvin Moon

Moon 83

Francis MoonGary Moon

Robert Charles Moon

Black Moon

Allan Moon

Ban-Ki Moon

Neil Armstrong

Eagle (lunar module)

Moon

Louis Armstrong

Fly me to the Moon (song)

Semantic AnalysisNamed Entity Mapping

„Armstrong landed the Eagle on the Moon.“

N. Steinmetz, H.Sack: Semantic Multimedia Information Retrieval Based on Contextual Descriptions, 2013

Entity Selection Process(Semantic) Graph Analysis

Montag, 26. Mai 14

4242 42 4224424242 42 4242

vfm - Seminar: Metadatenmanagement in Medienunternehmen, 05. September 2012, Bonn Jörg Waitelonis, Hasso-Plattner-Institut Potsdam

27

Semantically Annotated Video Data

Video Analysis /Metadata Extraction

timemetadata

metadatametadata

metadatametadata

e.g., person xylocation yzevent abc

e.g., bibliographical data,geographical data,encyclopedic data, ..

Entity Recognition/ Mapping

N. Ludwig, H. Sack: Named Entity Recognition for User-Generated Tags. In Proc. of the 8th Int. Workshop on Text-based Information Retrieval, IEEE CS Press, 2011

Montag, 26. Mai 14

Vorlesung Semantic Web, Dr. Harald Sack, Hasso-Plattner-Institut, Universität Potsdam

28

Entity Based Search

• linguistic ambiguities of traditional keyword based search can be avoided

• enables high precision and high recall retrieval

http://www.yovisto.com/labs/autosuggestion/

• Query string refinement / extension• entity auto-suggestion• interpretation of natural language queries

J. Osterhoff, J. Waitelonis, H. Sack, Widen the Peepholes! Entity-Based Auto-Suggestion as a rich and yet immediate Starting Point for Exploratory Search, IVDW 2012

Montag, 26. Mai 14

Vorlesung Semantic Web, Dr. Harald Sack, Hasso-Plattner-Institut, Universität Potsdam

29

http://semex.hpi.uni-potsdam.de/mggui-dev2

search facets

C. Hentschel, H. Sack, et al., Open up cultural heritage in video archives with mediaglobe, I2CS 2012

Montag, 26. Mai 14

Vorlesung Semantic Web, Dr. Harald Sack, Hasso-Plattner-Institut, Universität Potsdam

30

Towards Exploratory Search

dbpedia-owl:mission

dbpedia:Neil_Armstrong

dbpedia:Apollo_11dbpedia-owl:mission

category:Apollo_program

dcterms:subject

dbpedia:Apollo_13

dcterms:subject

yago:Space_accidents_and_incidents

rdf:type

rdf:type

dbpedia:Space_Shuttle_Challenger

dbpedia-owl:mission

J. Waitelonis, H. Sack: Towards exploratory video search using linked data, MTAP Volume 59, Number 2 (2012), 645-672

dbpedia:Buzz_Aldrin

dbpedia:Michael_Collins

Montag, 26. Mai 14

Exploratory Search and Serendipity•Find something that you were not looking for on purpose ...

dbpedia:Buzz_Aldrin

dbpedia:Cookie_Monster

dbpedia:Strictly_Come_Dancing

dbpedia:Transformers

Montag, 26. Mai 14

Explorative Search & Intelligent Recommmendationwith yovisto

J. Waitelonis, H. Sack: Towards exploratory video search using linked data, Multimedia Tools and Applications, Volume 59, Number 2 (2012), pp. 645-672

Montag, 26. Mai 14

Explorative Search & Intelligent Recommmendationwith yovisto

http://mediaglobe.yovisto.com:8080/

Montag, 26. Mai 14

Harald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, LDW 2011, Magdeburg, 30. Sep. 2011

Contact:Dr. Harald SackHasso-Plattner-Institut für SoftwaresystemtechnikUniversität PotsdamProf.-Dr.-Helmert-Str. 2-3D-14482 Potsdam

Homepage:http://www.hpi.uni-potsdam.de/meinel/team/sack.htmlBlog: http://yovisto.blogspot.com/E-Mail: harald.sack@hpi.uni-potsdam.de Twitter: lysander07 / biblionomicon / yovisto Slides can be found at http://slideshare.com/lysander07/

Thank you very much

for your attention!

Montag, 26. Mai 14