Upload
others
View
6
Download
0
Embed Size (px)
Citation preview
Lecture 4- !!!
Fei-Fei Li!
Lecture 4: Pixels and Filters
Professor Fei-‐Fei Li Stanford Vision Lab
25-‐Sep-‐13 1
Lecture 4- !!!
Fei-Fei Li!
What we will learn today?
• Images as funcFons • Linear systems (filters) • ConvoluFon and correlaFon
25-‐Sep-‐13 2
Some background reading: Forsyth and Ponce, Computer Vision, Chapter 7
Lecture 4- !!!
Fei-Fei Li!
• An image contains discrete number of pixels – A simple example – Pixel value:
• “grayscale” (or “intensity”): [0,255]
2010.12.18 3
Images as func,ons
231
75
148
Lecture 4- !!!
Fei-Fei Li!
• An image contains discrete number of pixels – A simple example – Pixel value:
• “grayscale” (or “intensity”): [0,255] • “color”
– RGB: [R, G, B] – Lab: [L, a, b] – HSV: [H, S, V]
2010.12.18 4
Images as func,ons
[249, 215, 203]
[90, 0, 53]
[213, 60, 67]
Lecture 4- !!!
Fei-Fei Li! 25-‐Sep-‐13 5
Images as func,ons • An Image as a funcFon f from R2 to RM:
• f( x, y ) gives the intensity at posiFon ( x, y ) • Defined over a rectangle, with a finite range:
f: [a,b] x [c,d ] à [0,255] Domain support
range
Lecture 4- !!!
Fei-Fei Li! 25-‐Sep-‐13 6
Images as func,ons • An Image as a funcFon f from R2 to RM:
• f( x, y ) gives the intensity at posiFon ( x, y ) • Defined over a rectangle, with a finite range:
f: [a,b] x [c,d ] à [0,255]
( , )( , ) ( , )
( , )
r x yf x y g x y
b x y
⎡ ⎤⎢ ⎥= ⎢ ⎥⎢ ⎥⎣ ⎦
• A color image:
Domain support
range
Lecture 4- !!!
Fei-Fei Li! 25-‐Sep-‐13 7
• Images are usually digital (discrete): – Sample the 2D space on a regular grid
• Represented as a matrix of integer values pixel
Images as discrete func,ons
Lecture 4- !!!
Fei-Fei Li! 25-‐Sep-‐13 8
Cartesian coordinates
Images as discrete func,ons
NotaFon for discrete funcFons
Lecture 4- !!!
Fei-Fei Li!
What we will learn today?
• Images as funcFons • Linear systems (filters) • ConvoluFon and correlaFon
25-‐Sep-‐13 9
Some background reading: Forsyth and Ponce, Computer Vision, Chapter 7
Lecture 4- !!!
Fei-Fei Li! 25-‐Sep-‐13 10
Systems and Filters • Filtering:
– Form a new image whose pixels are a combinaFon original pixel values
Goals: -‐ Extract useful informaFon from the images
• Features (edges, corners, blobs…)
-‐ Modify or enhance image properFes: • super-‐resoluFon; in-‐painFng; de-‐noising
Lecture 4- !!!
Fei-Fei Li! 25-‐Sep-‐13 11
Super-‐resoluFon De-‐noising
In-‐painFng
Bertamio et al
Lecture 4- !!!
Fei-Fei Li!
2D discrete-‐space systems (filters)
25-‐Sep-‐13 12
Lecture 4- !!!
Fei-Fei Li!
Filter example #1: Moving Average
• 2D DS moving average over a 3 × 3 window of neighborhood
25-‐Sep-‐13 13
1 1 1
1 1 1
1 1 1
],[],[91],)[(
,
lnkmhlkfnmhflk
−−=∗ ∑
h
Lecture 4- !!!
Fei-Fei Li! 25-‐Sep-‐13 14
0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 0 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 0 0 0 0 0 0 0
0 0 90 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0
0
0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 0 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 0 0 0 0 0 0 0
0 0 90 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0
Courtesy of S. Seitz ]ln,km[g]l,k[f]n,m)[gf(l,k
−−=∗ ∑
Filter example #1: Moving Average
Lecture 4- !!!
Fei-Fei Li! 25-‐Sep-‐13 15
15
0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 0 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 0 0 0 0 0 0 0
0 0 90 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0
0 10
0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 0 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 0 0 0 0 0 0 0
0 0 90 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0
]ln,km[g]l,k[f]n,m)[gf(l,k
−−=∗ ∑
Filter example #1: Moving Average
Lecture 4- !!!
Fei-Fei Li! 25-‐Sep-‐13 16
0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 0 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 0 0 0 0 0 0 0
0 0 90 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0
0 10 20
0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 0 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 0 0 0 0 0 0 0
0 0 90 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0
]ln,km[g]l,k[f]n,m)[gf(l,k
−−=∗ ∑
Filter example #1: Moving Average
Lecture 4- !!!
Fei-Fei Li! 25-‐Sep-‐13 17
0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 0 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 0 0 0 0 0 0 0
0 0 90 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0
0 10 20 30
0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 0 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 0 0 0 0 0 0 0
0 0 90 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0
]ln,km[g]l,k[f]n,m)[gf(l,k
−−=∗ ∑
Filter example #1: Moving Average
Lecture 4- !!!
Fei-Fei Li! 25-‐Sep-‐13 18
0 10 20 30 30
0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 0 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 0 0 0 0 0 0 0
0 0 90 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0
]ln,km[g]l,k[f]n,m)[gf(l,k
−−=∗ ∑
Filter example #1: Moving Average
Lecture 4- !!!
Fei-Fei Li! 25-‐Sep-‐13 19
0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 0 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 0 0 0 0 0 0 0
0 0 90 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0
0 10 20 30 30 30 20 10
0 20 40 60 60 60 40 20
0 30 60 90 90 90 60 30
0 30 50 80 80 90 60 30
0 30 50 80 80 90 60 30
0 20 30 50 50 60 40 20
10 20 30 30 30 30 20 10
10 10 10 0 0 0 0 0
]ln,km[g]l,k[f]n,m)[gf(l,k
−−=∗ ∑Source: S. Seitz
Filter example #1: Moving Average
Lecture 4- !!!
Fei-Fei Li! 25-‐Sep-‐13 20
In summary: • Replaces each pixel with an average of its neighborhood.
• Achieve smoothing effect (remove sharp features)
1 1 1
1 1 1
1 1 1
],[g ⋅⋅
Filter example #1: Moving Average
Lecture 4- !!!
Fei-Fei Li! 25-‐Sep-‐13 21
Filter example #1: Moving Average
Lecture 4- !!!
Fei-Fei Li! 25-‐Sep-‐13 22
• Image segmentaFon based on a simple threshold:
Filter example #2: Image SegmentaFon
Lecture 4- !!!
Fei-Fei Li!
ClassificaFon of systems
25-‐Sep-‐13 23
• Amplitude properFes • Linearity • Stability • InverFbility
• SpaFal properFes • Causality • Separability • Memory • Shih invariance • RotaFon invariance
Lecture 4- !!!
Fei-Fei Li!
Shih-‐invariance
25-‐Sep-‐13 24
If then
for every input image f[n,m] and shihs n0,m0
Lecture 4- !!!
Fei-Fei Li!
Is the moving average system is shih invariant?
25-‐Sep-‐13 25
0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 0 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 0 0 0 0 0 0 0
0 0 90 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0
0 10 20 30 30 30 20 10
0 20 40 60 60 60 40 20
0 30 60 90 90 90 60 30
0 30 50 80 80 90 60 30
0 30 50 80 80 90 60 30
0 20 30 50 50 60 40 20
10 20 30 30 30 30 20 10
10 10 10 0 0 0 0 0
Lecture 4- !!!
Fei-Fei Li!
Is the moving average system is shih invariant?
25-‐Sep-‐13 26
Yes!
Lecture 4- !!!
Fei-Fei Li!
Linear Systems (filters)
25-‐Sep-‐13 27
• Linear filtering: – Form a new image whose pixels are a weighted sum of original pixel values
– Use the same set of weights at each point
• S is a linear system (funcFon) iff it S sa&sfies
superposiFon property
Lecture 4- !!!
Fei-Fei Li!
• Is the moving average a linear system?
• Is thresholding a linear system? – f1[n,m] + f2[n,m] > T – f1[n,m] < T – f2[n,m]<T
25-‐Sep-‐13 28
Linear Systems (filters)
No!
Lecture 4- !!!
Fei-Fei Li!
LSI (linear shi$ invariant) systems
25-‐Sep-‐13 29
Impulse response
Lecture 4- !!!
Fei-Fei Li!
LSI (linear shi$ invariant) systems
25-‐Sep-‐13 30
1 1 1
1 1 1
1 1 1
h
Example: impulse response of the 3 by 3 moving average filter:
Lecture 4- !!!
Fei-Fei Li!
LSI (linear shi$ invariant) systems
25-‐Sep-‐13 31
An LSI system is completely specified by its impulse response.
superposiFon
Discrete convoluFon
sihing property of the delta funcFon
Lecture 4- !!!
Fei-Fei Li!
What we will learn today?
• Images as funcFons • Linear systems (filters) • ConvoluFon and correlaFon
25-‐Sep-‐13 32
Some background reading: Forsyth and Ponce, Computer Vision, Chapter 7
Lecture 4- !!!
Fei-Fei Li!
Discrete convolu,on
25-‐Sep-‐13 33
• Fold h[n,m] about origin to form h[−k,−l] • Shih the folded results by n,m to form h[n − k,m − l] • MulFply h[n − k,m − l] by f[k, l] • Sum over all k,l • Repeat for every n,m
h[k,l]
h[-‐k,-‐l] h[n-‐k,m-‐l]
n
Lecture 4- !!!
Fei-Fei Li!
Discrete convolu,on
25-‐Sep-‐13 34
• Fold h[n,m] about origin to form h[−k,−l] • Shih the folded results by n,m to form h[n − k,m − l] • MulFply h[n − k,m − l] by f[k, l] • Sum over all k,l • Repeat for every n,m
h[k,l]
h[-‐k,-‐l] h[n-‐k,m-‐l]
n
Lecture 4- !!!
Fei-Fei Li!
Discrete convolu,on
25-‐Sep-‐13 35
• Fold h[n,m] about origin to form h[−k,−l] • Shih the folded results by n,m to form h[n − k,m − l] • MulFply h[n − k,m − l] by f[k, l] • Sum over all k,l • Repeat for every n,m
h[n-‐k,m-‐l]
f[k,l]
f[k,l] x h[n-‐k,m-‐l]
Sum (f[k,l] x h[n-‐k,m-‐l])
n
Lecture 4- !!!
Fei-Fei Li!
Convolu,on in 2D -‐ examples
25-‐Sep-‐13 36
• 0 • 0 • 0 • 0 • 1 • 0 • 0 • 0 • 0
Original
? =
Courtesy of D
Low
e
Lecture 4- !!!
Fei-Fei Li!
Convolu,on in 2D -‐ examples
25-‐Sep-‐13 37
• 0 • 0 • 0 • 0 • 1 • 0 • 0 • 0 • 0
Original Filtered (no change)
=
Courtesy of D
Low
e
Lecture 4- !!!
Fei-Fei Li!
Convolu,on in 2D -‐ examples
25-‐Sep-‐13 38
• 0 • 0 • 0 • 1 • 0 • 0 • 0 • 0 • 0
Original
? =
Courtesy of D
Low
e
Lecture 4- !!!
Fei-Fei Li!
Convolu,on in 2D -‐ examples
25-‐Sep-‐13 39
• 0 • 0 • 0 • 1 • 0 • 0 • 0 • 0 • 0
Shifted right By 1 pixel
=
Courtesy of D
Low
e Original
Lecture 4- !!!
Fei-Fei Li!
Convolu,on in 2D -‐ examples
25-‐Sep-‐13 40
? • 1 • 1 • 1 • 1 • 1 • 1 • 1 • 1 • 1
=
Courtesy of D
Low
e Original
Lecture 4- !!!
Fei-Fei Li!
Convolu,on in 2D -‐ examples
25-‐Sep-‐13 41
• 1 • 1 • 1 • 1 • 1 • 1 • 1 • 1 • 1
Blur (with a box filter)
=
Courtesy of D
Low
e Original
Lecture 4- !!!
Fei-Fei Li!
Convolu,on in 2D -‐ examples
25-‐Sep-‐13 42
• 1 • 1 • 1 • 1 • 1 • 1 • 1 • 1 • 1
• 0 • 0 • 0 • 0 • 2 • 0 • 0 • 0 • 0 - = ?
(Note that filter sums to 1)
• 1 • 1 • 1 • 1 • 1 • 1 • 1 • 1 • 1
• 0 • 0 • 0 • 0 • 1 • 0 • 0 • 0 • 0 -
• 0 • 0 • 0 • 0 • 1 • 0 • 0 • 0 • 0
+
“details of the image”
Courtesy of D
Low
e
Original
Lecture 4- !!!
Fei-Fei Li! 25-‐Sep-‐13 43
• What does blurring take away?
original smoothed (5x5)
–
detail
=
sharpened
=
• Let’s add it back:
original detail
+ a
Lecture 4- !!!
Fei-Fei Li!
Convolu,on in 2D – Sharpening filter
25-‐Sep-‐13 44
• 1 • 1 • 1 • 1 • 1 • 1 • 1 • 1 • 1
• 0 • 0 • 0 • 0 • 2 • 0 • 0 • 0 • 0 -
Sharpening filter: Accentuates differences with local average
=
Courtesy of D
Low
e
Original
Lecture 4- !!!
Fei-Fei Li!
Convolu,on proper,es
25-‐Sep-‐13 45
• Commuta,ve property: • Associa,ve property: • Distribu,ve property:
The order doesn’t marer!
Lecture 4- !!!
Fei-Fei Li!
Convolu,on proper,es
25-‐Sep-‐13 46
• ShiI property: • ShiI-‐invariance:
Lecture 4- !!!
Fei-Fei Li!
Image support and edge effect
25-‐Sep-‐13 47
• A computer will only convolve finite support signals.
• That is: images that are zero for n,m outside some rectangular region
• MATLAB’s conv2 performs 2D DS convoluFon of finite-‐support signals.
N1 ×M1
N2 ×M2 (N1 + N2 − 1) × (M1 +M2 − 1) * =
Lecture 4- !!!
Fei-Fei Li!
Image support and edge effect
25-‐Sep-‐13 48
• A computer will only convolve finite support signals. • What happens at the edge?
f h
• zero “padding” • edge value replicaFon • mirror extension • more (beyond the scope of this class) -‐> Matlab conv2 uses zero-‐padding
Lecture 4- !!!
Fei-Fei Li!
What we will learn today?
• Images as funcFons • Linear systems (filters) • ConvoluFon and correlaFon
25-‐Sep-‐13 49
Some background reading: Forsyth and Ponce, Computer Vision, Chapter 7
Lecture 4- !!!
Fei-Fei Li!
Cross correla,on
25-‐Sep-‐13 50
Cross correlaFon of two 2D signals f[n,m] and g[n,m] • Equivalent to a convoluFon without the flip
(k, l) is called the lag
Lecture 4- !!!
Fei-Fei Li!
Cross correla,on – example
25-‐Sep-‐13 51
MATLAB’s xcorr2
Courtesy of J. Fessler
Lecture 4- !!!
Fei-Fei Li! 25-‐Sep-‐13 52
Cross correla,on – example Leh Right
scanline
Norm. corr
Lecture 4- !!!
Fei-Fei Li!
ConvoluFon vs. CorrelaFon
• A convolu,on is an integral that expresses the amount of overlap of one funcFon as it is shihed over another funcFon. – convoluFon is a filtering operaFon
• Correla,on compares the similarity of two sets of data. CorrelaFon computes a measure of similarity of two input signals as they are shihed by one another. The correlaFon result reaches a maximum at the Fme when the two signals match best . – correlaFon is a measure of relatedness of two signals
2010.12.18 53
Lecture 4- !!!
Fei-Fei Li!
Figure from “Computer Vision for Interactive Computer Graphics,” W.Freeman et al, IEEE Computer Graphics and Applications, 1998 copyright 1998, IEEE
Application: Vision system for TV remote control
- uses template matching
25-‐Sep-‐13 54
Lecture 4- !!!
Fei-Fei Li!
What we have learned today?
• Images as funcFons • Linear systems (filters) • ConvoluFon and correlaFon
25-‐Sep-‐13 55