Shuangyong Song, Qingliang Miao and Yao Meng | Linking Images to Semantic Knowledge Base with User-generated Tags

関係者外秘関係者外秘

Shuangyong Song

Fujitsu R&D Center Co., Ltd.

Linking Images to Semantic Knowledge Base with User-generated Tags

Copyright 2013 Fujitsu R&D Center Co.,LTD


Outlines

Motivation

Method

Experimental results

Conclusion

1Copyright 2010 FUJITSU LIMITED


Outlines

Motivation

Method


Conclusion



Motivation

Images account for an important part of Multimedia LinkedOpen Data. (LOD4ALL)

1. Wide semantic annotation of images with imageunderstanding is difficult and fallible; 2. semantic annotationwith manual labelling is time-consuming.

Therefore, automatic semantic annotation of images with usingother kind of crowdsourced ‘manual labelling’ is considered.


TagsSemantic

KnowledgeImages

medium

http://lod4all.net/statistics.html


Outlines

Motivation

Method


Conclusion



Method

Module 1. Linking tags to knowledge

Module 2. Detecting semantic relations between tags

Module 3. Extending knowledge linked tags

Module 4. Linking images to knowledge

Copyright 2013 Fujitsu R&D Center Co.,LTD5

[M1] [M2] [M3] [M4]

Semantic

Knowledge

Semantic

Knowledge

Semantic

Knowledge

Semantic

Knowledge

Semantic

Knowledge

Images Tags Topics


Method – Offline Part

Module 1. Linking tags to knowledge

Tag ‘Somophyllin-DF’: search it in dbo or dbr, and detect the unambiguous matched ontology as ‘dbr:Somophyllin-DF’,

‘Apple’, we may not be able to link it with ‘dbr:Apple_Inc.’ or ‘dbr:Apple_III’ or some other ontologies which related to ‘Apple’.


1 1

( , )* ( , )( , ) ( , , )

( )

a u u uU Ui k k ja u a u u

i j i k j uk k k

C t t C t tR t t R t t t

F t

http://dbpedia.org/resource/Apple_III



Module 2. Detecting semantic relations between tags


T DNd

θ(d)α

z

wϕ(z)β

Image data

Tags

Topic (vector)

Tags Topical vectors

For example, tags ‘programmer’ and ‘ilife’are detected as related tags.

Image-tag matrix is sparse and difficult to be well analyzed, topical information can help to discover the implicit semantic relations between tags.



Module 3. Extending knowledge linked tags

The rules are as below: (very strict for keep precision)

1) If t1 and t2 have ‘inclusion relation’, such as ‘motor’ and ‘motorcycle’, and cosine-similarity value between their topical vectors is bigger than σ, we judge them as similar tags;

2) If ‘Levenshtein distance’ value between t1 and t2 is equal to or smaller than a threshold β, which is a positive integer, and cosine-similarity value between their topical vectors is bigger than σ, we judge them as similar tags;

3) If t1 and t2 don’t have above relations, we judge them as dissimilar tags.

If t1 and t2 are judged as similar tags, and just one of them has semantic link to knowledge base, we link the other one to the same knowledge with a probability, of which the value equals to C(t1,t2), which is the cosine-similarity value between t1 and t2.



Method – Online Part

Module 4. Linking images to knowledge


Define the linking types <subject, predicate, object>

We use 77 million triples collected from our semantic database ‘LOD4ALL’ as

criteria for predicate selection. We create links between images and objects with

unambiguous predicate. For example, we use predicate ‘dbo:locationCountry’ to

create links to objects such as ‘China’ or ‘Denmark’.

We use ‘rdfs:seeAlso’ to be predicate for ambiguous triples.

In particular, we also detect some tag-combined knowledge for expanding the

links’ range. For example, if an image has both ‘DigitalCamera’ and ‘Conon’ as its

tags, we will check if there is ‘DigitalCamera_Conon’ or ‘Conon_ DigitalCamera’

in dbo or dbr, and link this image to the detected knowledge.


Outlines

Motivation

Method


Conclusion




1. Experimental Datasets

2. Methods for Comparison

3. Evaluation Metrics & Test Data Preparation

4. Evaluation Results





mirFlickr 1 , 690,649 images and 659,227 tags.

all the 7,401 ontologies in dbo and 7,925,232 resources in dbr were used as semantic KB.

After removing 41,127 images without any tag and 392 attached with ‘stopword’ tags, we had 649,130 images left, and those images and their tags were used to train the topic model with an empirical K value of 200.


1 Huiskes, M. J., Lew, M. S. The MIR Flickr Retrieval Evaluation. In: Proceedings of the 1st ACM SIGMMInternational Conference on Multimedia Information Retrieval, 2008, pp.39-43.




2. Methods for Comparison (our model: LDA based)


Word2vec based

model (W2V)

Word2vec is a recently popular method for getting distributedrepresentation of words, of which the output form is similar toLDA. However, we suppose that W2V is unfit for the image-knowledge linking task, since W2V does well in detecting contextsimilarity, such as the similarity between ‘Paris’ and ‘Beijing’.We will evaluate it in the following part.

Co-occurrence based

model (Co-occur)

Co-occur is a very simple yet effective method for detectingrelationship between words, we also take it as a baseline onthe tag similarity calculation subtask instead of LDA methodin subsection 2.3, and evaluate it in in the following part.Compared to our model, Co-occur is a weaker tool on detectinglatent semantic information of tags.







To evaluate the effects of different d when β = 2, we manually label 200 images.

We roughly set the recall as 1.0 when d = 1 considering we are unable to collect

all possible related knowledge to an image. After getting both precision and

recall with different d, we also use F1-value as evaluation criterion to evaluate d.

precision ; recall ; F1-value.






4. Evaluation Results


car girl flower bird dog sport

W2V 0.217 0.531 0.233 0.389 0.325 0.442

Co-occur 0.683 0.724 0.631 0.557 0.626 0.702

Our model 0.881 0.767 0.854 0.871 0.873 0.801

Result comparison with F1-value


Experimental results - examples


schema:Brand

dbr:Citroen_XM

dbr:French

rdfs:seeAlso

schema:inLanguagedbr:Citroen

dbr:Car

schema:category

schema:location

dbr:Mallorca

dbr:Palma

schema:location

schema:location

dbr:Baleares

dbr:Sineu

schema:location

dbr:Geotagged

schema:indication


Outlines

Motivation

Method


Conclusion



Conclusion

Summary:

In this paper, we address the problem of automatically linking

images to semantic KB by using images’ tagging information. Our

framework lifted the F1-value of images’ semantic link creation by

62.43%.

Future works:

However, this paper is only a preliminary work, and named entity

disambiguation and named entity normalization will be considered

in our next step work. Besides, predicate discriminate should be

performed by considering adjacent tags as context. In addition, we

will try to create ground-truth datasets and try some classification

models on this task.



Thank you！


関係者外秘関係者外秘 20 Copyright 2013 Fujitsu R&D Center Co.,LTD

Technology

Shuangyong Song, Qingliang Miao and Yao Meng | Linking Images to Semantic Knowledge Base with User-generated Tags