Embed Size (px)
Citation preview
ECE160 Spring 2009 Lecture 20 1
ECE160 / CMPS182
Multimedia
Lecture 20: Spring 2009Image Recognition and
Retrieval
ECE160Spring 2009 Image Recognition 2
National Research Priorities Energy Technologies
Fuel efficient engines Replacement energy to fossil fuels Lighter, longer-duration batteries
Bioengineering/Bioinformatics Genes disease Disease medicine
Search with Multimedia Content Video surveillance Photo interpretation
ECE160Spring 2009 Image Recognition 3
Multimedia Recognition Video surveillance Photo interpretation
ECE160Spring 2009 Image Recognition 4
Wide-area Surveillance
advertisement of objectvideo.com
ECE160Spring 2009 Image Recognition 5
Surveillance Scenarios(1) Intrusion Detection(1) Intrusion Detection (2) Passenger Screening(2) Passenger Screening
(3) Perimeter Monitoring(3) Perimeter Monitoring
ZZ
Use biometric facial recognition to identify individuals of interest through existing closed circuit TV surveillance
Monitor and alert on tailgating, loitering, exit/closed entry, other unauthorized access
Object tracking and biometric facial recognition to determine vehicles and humans exhibiting suspicious behavior
(4) Unattended Baggage(4) Unattended Baggage
Copyright © 2004 Proximex Corp.Copyright © 2004 Proximex Corp.
Identify unattended baggage (or other objects) left for long periods of time
ECE160Spring 2009 Image Recognition 6
Multimedia Recognition Video surveillance Photo interpretation
ECE160Spring 2009 Image Recognition 7
How to Organize these Photos?
ECE160Spring 2009 Image Recognition 8
Keyword-based Manual labeling is subjective,
cumbersome The aliasing problem
Content-based Promising for general semantics:
outdoor, landscape, flowers, people, etc. Not enough for wh-queries (where, who,
when, or what)
Image Organization & Retrieval
ECE160Spring 2009 Image Recognition 9
EXTENTTM = contEXT + contENT
Context Spatial (location) Temporal Social Others
Content Perceptual features, such as color,
texture, and shape Holistic features and local features
ECE160Spring 2009 Image Recognition 10
EXTENTTM
Content
Spatial Temporal
Social Others
ECE160Spring 2009 Image Recognition 11
Augmented Images
+
Cameraphones with high-quality lens can record location, time, camera parameters, and voice
=
ECE160Spring 2009 Image Recognition 12
Context from Space/Time
GPS or CellID data Into place names
Time-based grouping Into meaningful “events”
From place names and time Time of day Weather
ECE160Spring 2009 Image Recognition 13
Example of Using Three Pieces of
Information
Content
Spatial Temporal
ECE160Spring 2009 Image Recognition 14
Maui Sunsetscan be obtained from Space/Time
ECE160Spring 2009 Image Recognition 15
Use content for verification
ECE160Spring 2009 Image Recognition 16
Use content to transfer metadata
ECE160Spring 2009 Image Recognition 17
Summarize of the example
Derived from Context Derive time of the day Obtain weather Verify content
Use of Content Verify context Transfer context
Much more…
ECE160Spring 2009 Image Recognition 18
Are They Similar?
ECE160Spring 2009 Image Recognition 19
Are They Similar?
ECE160Spring 2009 Image Recognition 20
Are They Similar?
In terms of what? What is the user’s perception?
ECE160Spring 2009 Image Recognition 21
Conveying Perception
Image Databases Conveyed via Examples
Use a sunset picture (or pictures) to find more sunset images
Where does the perfect example come from?
ECE160Spring 2009 Image Recognition 22
Conveying Perception
Internet Searches Conveyed via Keywords
ECE160Spring 2009 Image Recognition 23
Keyword Retrieval
Pros A user-friendly paradigm
Cons Annotation is a laborious process Annotation quality can be subpar Annotation can be subjective Synonyms
ECE160Spring 2009 Image Recognition 24
Conveying Perception
Image Databases Conveyed via Examples
Use a sunset picture (or pictures) to find more sunset images
Where does the perfect example come from?
ECE160Spring 2009 Image Recognition 25
Are They Similar?
ECE160Spring 2009 Image Recognition 26
Are They Similar?
ECE160Spring 2009 Image Recognition 27
Are They Similar?
In terms of what? What is the user’s perception?
ECE160Spring 2009 Image Recognition 28
Recogintion of Content
ECE160Spring 2009 Image Recognition 29
Recognition
ECE160Spring 2009 Image Recognition 30
Recognition
ECE160Spring 2009 Image Recognition 31
clouds vs. waves
ECE160Spring 2009 Image Recognition 32
Web 1.0 vs. Web 2.0
User
User User
Content
Content (Image/Video)
Content
Content
Content
Content
Content
Content (text)
Content
Content
Content
Content
Content
User
UserUser User
ECE160Spring 2009 Image Recognition 33
Web 2.0 Content + Users + Interactions Collect rich, organized content Attract users & interactions
To provide metadata To provide new content
Improve search quality With new metadata and data Via social-network structure
ECE160Spring 2009 Image Recognition 34
User management
User management
Photo uploading
Single/multiple
Upload wizard
Photo uploading
Building social network
Social networks
Event management
Event management
Photo search
Photo search
Metadata collectionMetadata fusionImage annotation
Metadata collection - contextual - content
Metadata fusion
Annotate photos
External functionalities
Internal functionalities
Fotofiti
