Chapter 5 Image Restoration

Chapter 5Image Restoration

國立雲林科技大學電子工程系張傳育 (Chuan-Yu Chang ) 博士Office: ES 709TEL: 05-5342601 ext. 4337E-mail: [email protected]

2

Chapter 5 Image Restoration Image Degradation/Restoration Process

The objective of restoration is to obtain an estimate of the original image.

will be close to f(x,y).

),(ˆ yxf

),(ˆ yxf

Restoration 的目的在於獲得原始影像的估計影像，此估計影像應儘可能

的接近原始影像。

3

Image Degradation/Restoration Process

The degraded image is given in the spatial domain by

The degraded image is given in the frequency domain by

),(),(*),(),( yxyxfyxhyxg

),(),(),(),( vuNvuFvuHvuG

(5.1-1)

(5.1-2)

Degradation function

noise

4

Noise Models: Some Important Probability Density Functions The principal sources of noise

Image acquisition transmission

Gaussian noise (normal noise)

Rayleigh noise

Erlang (Gamma) noise

22 2/)(1

2

1)(

zezp

4

)4(

4/

0

)(2

)(

2

/)( 2

b

ba

azfor

azforeazbzp

baz

22

1

00

0)!1()(

a

ba

b

zfor

zforeb

zazp

azbb

5

Noise Models

Exponential noise

Uniform noise

Impulse (salt and pepper) noise12

)(

2

0

1)(

22 ab

ba

otherwise

bzaifabzp

otherwise

bzforP

azforP

zp b

a

0

)(

22 1

1

00

0)(

a

a

zfor

zforaezp

az

6

Some important probability density function

偏離原點的位移，向右傾斜

7

Example 5.1Sample noisy images and their histograms 下圖為原始影像，將在此圖中加入不同的雜訊。

8

Example 5.1 (cont.)Sample noisy images and their histograms

9

Example 5.1 (cont.) Sample noisy images and their histograms

10

Periodic Noise

Periodic Noise Arises typically from electrical or electromechanical

interference during image acquisition. It can be reduced via frequency domain filtering.

Estimation of Noise Parameters Estimated by inspection of the Fourier spectrum of the

image. Periodic noise tends to produce frequency spikes that often

can be detected by visual analysis. From small patches of reasonably constant gray level. The heights of histogram are different but the shapes are

similar.

11

Example影像遭受sinusoidal

noise 的破壞

影像的spectrum

有規則性的亮點

12

Fig 5.4(a-c) 中的某小塊影像的histogram

Histogram 的形狀幾乎和 Fig4(d,e,k) 的形狀一樣但高度不同。

13

Periodic Noise

The simplest way to use the data from the image strips is for calculating the mean and variance of the gray levels.

The shape of the histogram identifies the closest PDF match.

Sz

iii

zpz

Sz

iii

zpz 22

(5.2-15)

(5.2-16)

14

Restoration in the presence of noise only-spatial filtering Degradation present in an image is noise

The noise terms ((x,y), N(u,v)) are unknown, so subtracting them from g(x,y)or G(u,v)is not a realistic option.

In periodic noise ， it is possible to estimate N(u,v) from the spectrum of G(u,v).

),(),(),(

),(),(),(

vuNvuFvuG

yxyxfyxg

15

Mean Filter Arithmetic mean filter

Let Sxy represent the set of coordinates in a rectangular subimage windows of size mxn, centered at point (x,y).

The arithmetic mean filtering process computes the average value of the corrupted image g(x,y) in the area defined by Sxy.

This operation can be implemented using a convolution mask in which all coefficients have value 1/mn.

Noise is reduced as a result of blurring

xySts

tsgmn

yxf),(

),(1

),(ˆ

16

Mean Filter (cont.)

Geometric mean filter Each restored pixel is given by the product of the pixels in

the subimage window, raised to the power 1/mn.

A geometric mean filter achieves smoothing comparable to the arithmetic mean filter, but it tends to lose less image detail in the process.

mn

Sts xy

tsgyxf

1

),(

),(),(ˆ

17

Example 5.2Illustration of mean filters

被平均值被平均值 00 ，變異數，變異數400400 的加成性高斯雜的加成性高斯雜

訊破壞的結果訊破壞的結果

18

Restoration in the presence of noise only-spatial filtering

Harmonic mean filter

Contra-harmonic mean filter

xySts tsg

mnyxf

),( ),(1

),(ˆ

xy

xy

Sts

Q

Sts

Q

tsg

tsg

yxf

),(

),(

1

),(

),(

),(ˆ

可濾除 salt noise,但對 pepper noise 則失

敗

若 Q>0 可濾除 pepper noise,

若 Q<0 可濾除 salt noise,若 Q=0 為算數平均若 Q=-1 為 Harmonic mean

19



被機率被機率 0.10.1 的的 saltsalt 雜雜訊破壞的結果訊破壞的結果

被機率被機率 0.10.1 的的 pepperpepper雜訊破壞的結果雜訊破壞的結果

The positive-order filter did a better job of cleaning the background.In general, the arithmetic and geometric mean filters are well suited for random noise.The contraharmonic filter is well suited for impulse noise

20

Results of selecting the wrong sign in contra-harmonic filtering

The disadvantage of contraharmonic filter is that it must be known whether the noise is dark or light in order to select the proper sign for Q. The result of choosing the wrong sign for Q can be disastrous.在 contra-harmonic filter 中選取錯誤正負號所導致的結果

21

Order-Statistics Filters The response of the order-statistics filters is based

on ordering (ranking) the pixels contained in the image area encompassed by the filter.

Median filter Replaces the value of a pixel by the median of the gray

levels in the neighborhood of that pixel.

Medial filter provide excellent noise-reduction capabilities, with considerably less blurring than linear smoothing filters of similar size.

Median filters are particularly effective in the presence of both bipolar and unipolar impulse noise.

tsgmedianyxfxySts

,),(ˆ),(

(5.3-7)

22

Order-Statistics Filters (cont.) Max filter

This filter is useful for finding the brightest points in an image.

It reduces pepper noise

Min filter This filter is useful for finding the darkest points in an image

It reduces salt noise.

tsgyxfxySts

,max),(ˆ),(

tsgyxfxySts

,min),(ˆ),(

可濾除 pepper noise

可濾除 salt noise

(5.3-8)

(5.3-9)

23

Order-Statistics Filters (cont.) Midpoint filter

This filter works best for randomly distributed noise, such as Gaussian or uniform noise.

Alpha-trimmed mean filter We delete the d/2 lowest and the d/2 higest gray-level values

of g(s,t) in the neighborhood Sxy.

tsgtsgyxf

xyxy StsSts,min,max

2

1),(ˆ

),(),(

xySts

r tsgdmn

yxf),(

),(1

),(ˆ(5.3-11)

(5.3-10)

先刪除 0.5d 最大與最小的灰階值，再求剩下的灰階平均值當 d=0 ，則為 mean filter當 d=(mn-1)/2 時，為 median filter

24

Example 5.3Illustration of order-statistics filtersImage corrupted by salt

and pepper noise with probabilities Pa=Pb=0.1

Result of one pass with a median filter of size 3x3, several noise points are still visible.

Result of processing (b) with median filter again

Result of processing (c) with median filter again

25

Example 5.3Illustration of order-statistics filters

Result of filtering with a max filtering

Result of filtering with a min filtering

26

Example 5.3 Illustration of order-statistics filters

Result of filtering with a arithmetic

mean filter

Result of filtering with a geometric

mean filter

Result of filtering with a median filter

Result of filtering with a alpha-

trimmed mean filter

27

Adaptive Filter Adaptive Filter

The behavior changes based on statistical characteristics of the image inside the filter region defined by the m x n rectangular windows Sxy.

The price paid for improved filtering power is an increase in filter complexity.

Adaptive, local noise reduction filter The mean gives a measure of average gray level in the

region. The variance gives a measure of average contrast in that

region. The response of the filter at any point (x,y) on which the

region is centered is to be based on four quantities: g(x,y): the value of the noisy image. The variance of the noise corrupting f(x,y) to form g(x,y) mL, the local mean of the pixels in Sxy. The local variance of the pixels in Sxy.

28

Adaptive local noise reduction filter The behavior of the filter to be as follows:

If the variance of g(x,y) is zero, the filter should return simply the value of g(x,y).

If the local variance is high relative to the variance of g(x,y) , the filter should return a value close to g(x,y).

If the two variances are equal, return the arithmetic mean value of the pixels in Sxy.

An adaptive expression for obtaining estimated f(x,y) based on these assumptions may be written as

LL

myxgyxgyxf ,),(),(ˆ2

2

(5.3-12)

The only quantity thatNeeds to be known

29

Example 5.4 Illustration of adaptive, local noise-reduction filtering

Gaussian noise

Arithmetic mean 7*7

geometic mean 7*7

Adaptive filter

30

Adaptive median filter

Adaptive median filtering can handle impulse noise, it seeks to preserve detail while smoothing nonimpulse noise.

The adaptive median filter changes the size of Sxy during filter operation, depending on certain conditions.

Consider the following notation: Zmin: minimum gray level value in Sxy. Zmax: maximum gray level value in Sxy. Zmed: median of gray levels in Sxy. Zxy: gray level at coordinates (x,y). Smax: maximum allowed size of Sxy.

31

Adaptive median filter (cont.) The adaptive median filtering algorithm Level A: A1=zmed-zmin

A2=zmed-zmax

if A1>0 and A2<0, goto level Belse increase the window sizeif window size <=Smax, repeat level

Aelse output zxy

Level B: B1=zxy-zmin

B2=zxy-zmax

if B1>0 and B2 <0, output zxy

else output zmed

判斷 zmed 是否為impulse noise

32

Adaptive median filter (cont.)

The objectives of the adaptive median filter Remove the slat-and-pepper noise Preserve detail while smoothing nonimpulse noise Reduce distortion

The purpose of level A is to determine in the median filter output, zmed is an impulse or not.

33

Example 5.5 Illustration of adaptive median filtering

Corrupted by salt-and pepper noise with probabilities

Pa=Pb=0.25

Result of filtering with a 7x7 median filter

Result of adaptive median filtering with

Smax=7

The noise was effectively removed, the filter caused significant loss of detail in the image

Preserved sharpness and detail

34

Periodic Noise Reduction by Frequency Domain Filtering Bandreject Filter

Remove a band of frequencies about the origin of the Fourier transform.

2),(1

2),(

20

2),(1

),(

0

00

0

WDvuDf

WDvuD

WDif

WDvuDif

vuH

n

DvuDWvuD

vuH 2

20

2 ),(),(

1

1),(

220

2

),(

),(

2

1

1),(

WvuD

DvuD

evuH

Ideal Bandreject filter

N order Butterworth filter

Gaussian Bandreject filter

(5.4-1)

(5.4-2)

(5.4-3)

35

Periodic Noise Reduction by Frequency Domain Filtering (cont.)

36

Image corrupted by sinusoidal noise

Butterworth bandreject filter of order 4


37


Bandpass Filters A bandpass filter performs the opposite operation of a

bandreject filter. The transfer function Hbp(u,v) of a bandpass filter is

obtained from a corresponding bandreject filter with transfer function Hbr(u,v) by

),(1),( vuHvuH brbp (5.4-4)

38

以帶通濾波器所獲得的圖 5.16(a) 影像的

雜訊圖樣


Bandpass filtering is quit useful in isolating the effect on an image of selected frequency bands.

This image was generated by(1) Using Eq(5.4-4) to obtain

the bandpass filter.(2) Taking the inverse

transform of the bandpass-filtered transform

39

Periodic Noise Reduction by Frequency Domain Filtering (cont.) Notch Filters

Rejects frequencies in predefined neighborhoods about a center frequency.

Due to the symmetry of the Fourier transform, notch filters must appear in symmetric pairs about the origin

212

02

02

2

12

02

01

0201

2/2/),(

2/2/),(

1

),(),(0),(

vNvuMuvuD

vNvuMuvuD

otherwise

DvuDorDvuDifvuH

(5.4-5)

(5.4-6)

(5.4-7)

因為對稱性的關係

40


order n Butterworth notch filter

Gaussian notch reject filter

These three filters become highpass filters if u0=v0=0.

n

vuDvuDD

vuH

),(),(1

1),(

21

20

(5.4-8)

20

21 ),(),(

2

1

1),( D

vuDvuD

evuH (5.4-9)

41

Ideal notch

order 2 Butterworth notch filter

Gaussian notch filter

若 u0=v0=0 ，上述三種濾波器，則退化成高通濾波器


42


Notch pass filters We can obtain notch pass filters that pass the frequencies

contained in the notch areas. Exactly the opposite function as the notch reject filters.

Notch pass filters become lowpass filters when u0=v0=0.

vuHvuH nrnp ,1, (5.4-10)(5.4-10)

43

佛羅里達州和墨西哥灣的衛星影像

( 存在水平掃描線 )

Spectrum image

Notch filter

空間域的雜訊影像


44


Optimal Notch filtering Clearly defined interference patterns are not

common. Images obtained from electro-optical scanner are

corrupted by coupling and amplification of low-level signals in the scanners’ electronic circuitry.

The resulting images tend to contain significant, 2D periodic structures superimposed on the scene data.

45


Image of the Martian terrain taken by the Mariner 6 spacecraft. The interference pattern is hard to detect. The star-like components were caused by the interference, and

several pairs of components are present. The interference components generally are not single-frequency

bursts. They tend to have broad skirts that carry information about the interference pattern.

46


Optimal Notch filtering minimizes local variances of the restored estimate image.

The procedure contains three steps: Extract the principal frequency components of the

interference pattern. Subtracting a variable, weighted portion of the

pattern from corrupted image.

47


The first step is to extract the principal frequency component of the interference pattern Done by placing a notch pass filter, H(u,v) at the

location of each spike. The Fourier transform of the interference noise

pattern is given by the expression

where G(u,v) denotes the Fourier transform of the corrupted image.

vuGvuHvuN ,,,

48


Formation of H(u,v) requires considerable judgment about what is or is not an interference spike. The notch pass filter generally is constructed

interactively by observing the spectrum of G(u,v) on a display.

After a particular filter has been selected, the corresponding pattern in the spatial domain is obtained from the expression

vuGvuHyx ,,, 1

49


Because the corrupted image is assumed to be formed by the addition of the uncorrupted image f(x,y) and the interference, if (x,y) were know completely, subtracting the pattern from g(x,y) to obtain f(x,y) would be a simple matter. This filtering procedure usually yields only an approximation of

the true pattern. The effect of components not present in the estimate of (x,y)

can be minimized instead by subtracting from g(x,y) a weighted portion of (x,y) to obtain an estimate of f(x,y).

The function w(x,y) is to be determined, which is called as weighting or modulation function.

The objective of the procedure is to select this function so that the result is optimized in some meaningful way.

yxyxwyxgyxf ,,,,ˆ (5.4-13)

50


To select w(x,y) so that the variance of the estimate f(x,y) is minimized over a specified neighborhood of every point (x,y).

Consider a neighborhood of size (2a+1) by (2b+1) about a point (x,y), the local variance can be estimated as

where

a

as

b

bt

yxftysxfba

yx2

2 ,ˆ,ˆ1212

1,

a

as

b

bt

tysxfba

yxf ,ˆ1212

1,ˆ

(5.4-15)

(5.4-14)

51


Substituting Eq(5.4-13) into Eq(5.4-14) yield

Assuming that w(x,y) remains essentially constant over the neighborhood gives the approximation

This assumption also results in the expression

in the neighborhood.

),(, yxwtysxw

22 ,,),(,,,1212

1, yxyxwyxgtysxtysxwtysxg

bayx

a

as

b

bt

yxyxwyxyxw ,,,, (5.4-18)

(5.4-17)

(5.4-16)

52


With these approximations Eq5.4-160 becomes

To minimize variance, we solve

for w(x,y)

The result is

22 ,,),(,,,1212

1, yxyxwyxgtysxtysxwtysxg

bayx

a

as

b

bt

0

,

,2

yxw

yx

),(),(

),(,),(,,

22 yxyx

yxyxgyxyxgyxw

(5.4-21)

(5.4-20)

(5.4-19)

53

圖 5-20(a) 的傅立葉頻譜


54

N(u,v) 的傅立葉頻譜

對應的雜訊圖樣


55

處理後的影像


56

Linear, Position-Invariant Degradations

)],([)],([)],(),([

1

)],([)],([)],(),([

)],([),(

0),(

),()],([),(

2121

2121

yxfHyxfHyxfyxfH

baIf

yxfbHyxfaHyxbfyxafH

iflinearisH

yxfHyxg

yxthatassume

yxyxfHyxg

AdditivityIf H is a linear operator, the response to a sum of two inputs is equal to the sum of the two response

(5.5-1)

(5.5-2)

(5.5-3)

57


Homogeneity The response to a constant multiple of any input is equal

to the response to that input multiplied by the same constant.

),(),(

)2.5.5.(,0),(

11

2

yxfaHyxafH

becomesEqyxfwith

(5.5-4)

58


Position (space) invariance The response at any point in the image depends only on the

value of the input at that point, not on its position.

f(x,y) 可用連續脈衝函數表示成

假設 (x,y)=0, 則將 Eq(5.5-6) 代入 Eq(5.5-1) 可得

因為 H 為線性運算子 , 利用加成性的性質

),(),( yxgyxfH (5.5-5)

ddyxfyxf

,,,

ddyxfHyxfHyxg ,,,,

(5.5-6)

(5.5-7)

ddyxfHyxfHyxg

,,,, (5.5-8)

59

Linear, Position-Invariant Degradations (cont.)

又因為 f(,) 和 x,y 無關 , 因此利用 Homogeneity 可得

其中 ,H 為脈衝響應 (impulse response),h(x,,y,) 為點展開函數 (point spread function, PSF)

將 Eq(5.5-10) 代入 Eq(5.5-9) 可得

ddyxHfyxfHyxg

,,,,

yxHyxh ,,,,

(5.5-9)

(5.5-10)

ddyxhfyxg

,,,,, (5.5-11)

60

Linear, Position-Invariant Degradations (cont.) 若 H 為位置不變 , 則 Eq(5.5-5) 可知

則 Eq(5.5-11) 可變成

上式為 convolution integral( 同 Eq(4.2-30)) 若在有加成性雜訊下 , Eq(5.5-11) 可表示成

若 H 是位置不變 , 則 Eq(5.5-14) 會變成

yxhyxH ,,

ddyxhfyxg

,,,

yxddyxhfyxg ,,,,,,

yxddyxhfyxg ,,,,

(5.5-12)

(5.5-13)

(5.5-14)

(5.5-15)

61

Linear, Position-Invariant Degradations (cont.) Summary

因為雜訊項 (x,y) 是隨機 , 與位置無關的 , 可將 Eq(5.5-15) 改寫成

A linear, spatially invariant degradation system with additive noise can be modeled in the spatially domain as the convolution of the degradation function with an image, followed by the addition of noise.

),(),(),(),(

),(),(*),(),(

vuNvuFvuHvuG

yxyxfyxhyxg

(5.5-16)

(5.5-17)

62

Estimating the Degradation Function

There are three principal ways to estimate the degradation function for use in image restoration: Observation Experimentation Mathematical modeling

63


Estimation by image observation When a given degraded image without any knowledge about the

degradation function H. To gather information from the image itself.

Look at a small section of the image containing simple structures. Look for areas of strong signal content. Gs(u,v) Construct an unblurred image as the observed subimage. Fs(u,v) Assume that the effect of noise is negligible, thus the degradation

function could be estimated by Hs(u,v)=Gs(u,v)/Fs(u,v) To construct the function H(u,v) by turns out the Hs(u,v) to have the

shape of Butterworth lowpass filter.

64


Estimation by experimentation A linear, space-invariant system is described completely

by its impulse response. A impulse is simulated by a bright dot of light 把一個已知的函數加以模糊以便得到近似的 H(u,v)

A

vuGvuH

vuFvuHvuG

),(),(

1)y)(x,(constant aisimpulseanoftransformFourierthe

),(),(),(

(5.6-2)

65

光脈衝影像脈衝


66

Estimating the Degradation Function Estimation by modeling

Degradation model based on the physical characteristics of atmospheric turbulence:

6/522 )(),( vukevuH

67


68

Remove the degradation of planar motion

)(

0

/2

0

)(2

0

0

)]()([2

0

)]()([2

)]()([2

0

)(2

0 00

)(2

0 00

)](sin[)(

),(

)sin(),(

/)(

by given rate aat

onlydirection - xin themotion linear uniform undergoes image that thesuppose

),(),(),(

),(

),(

),(

)(),(

),(),(

is 4)-Eq.(5.6 of TransformFourier the

)(),(),(

0

00

00

00

vbuaj

uajT TuatjT tuxj

T tvytuxj

T tvytuxj

tvytuxjT

vyuxjT

vyuxj

T

evbuavbua

TvuH

euaua

TdtedtevuH

Tattx

vuFvuHvuG

dtevuH

dtevuF

dtevuF

dxdyedttyytxxf

dxdyeyxgvuG

dttyytxxfyxg

(5.6-11)

(5.6-10)

(5.6-9)(5.6-8)

69


Chapter 5Image Restoration 將左圖的 Fourier

Transform 乘上 (5.6-11)的 H(u,v) 後，取其反

Fourier Transform 的結果。a=b=0.1, T=1

70

Inverse Filtering

Direct inverse filtering 根據 Eq(5.1-2) 直接進行反濾波

即使知道退化函數 , 也無法完成復原出無退化的影像 , 因為N(u,v) 為一未知的傅立葉轉換隨機函數

假使退化有零值會是非常小的數值 , 則 N(u,v)/H(u,v) 會嚴重影響 F(u,v)

),(

),(),(),(ˆ

),(),(),(),(

),(

),(),(ˆ

vuH

vuNvuFvuF

vuNvuFvuHvuG

vuH

vuGvuF

(5.7-1)

(5.7-2)

71



直接G(u,v)/H(u,v)

Cutoff H(u,v) a radius of 40

Cutoff H(u,v) a radius of 70

Cutoff H(u,v) a radius of

85

72

Minimum Mean Square Error (Wiener) Filtering Incorporated both the degradation function and statistical

characteristics of noise into the restoration process. The objective is to find an estimate f of the uncorrupted

image f such that the mean square error between them is minimized.

),(),(

),(

),(

1

),(),(/),(),(

),(

),(

1

),(),(/),(),(

),(

),(),(),(),(

),(),(),(ˆ

ˆ

2

2

2

2

2

*

2

*

22

vuGKvuH

vuH

vuH

vuGvuSvuSvuH

vuH

vuH

vuGvuSvuSvuH

vuH

vuGvuSvuHvuS

vuSvuHvuF

ffEe

f

f

f

f

(5.8-1)

(5.8-2)

(5.8-3)

通常為未知，因此以 K 來估計

73

Example 5.12

74

Example 5.13

75

Constrained Least Squares Filtering

The difficulty of the Wiener filter: The power spectra of the undegraded image and noise

must be known A constant estimate of the ratio of the power spectra is not

always a suitable solution. Constrained Least Squares Filtering

Only the mean and variance of the noise are needed.

76

Constrained Least Squares Filtering

010

141

010

),(

),(),(),(

),(),(ˆ

ˆ

constraint thesubject to

),(

22

*

22

21

0

1

0

2

yxp

vuGvuPvuH

vuHvuF

yxfCM

x

N

y

fHg

ηHfg

將 Eq(5.5-16) 以向量矩陣表示成(5.9-1)

(5.9-2)

(5.9-3)

(5.9-4)

(5.9-5)

77



手動調整 r ，以獲得最佳視覺效果

78

以遞迴方式計算定義殘餘向量

fHgr ˆ

2rrr T

a 22ηr

79

Step 1: 指定的初始值 Step 2: 計算 ||r||2

Step 3: 若滿足 Eq(5.9-8) 即停止 ,若即增加的值若即減少的值回到步驟 2, 使用新的值計算 Eq(5.9-4)

a 22ηr

a 22ηr

80



81

Geometric Transformations Spatial Transformations

8765

4321

8765

4321

'

'

'

'

),(

),(

),(

),(

cxycycxcy

cxycycxcx

cxycycxcyxs

cxycycxcyxr

yxsy

yxrx

82



Gray-level Interpolation Zero-order interpolation Cubic convolution interpolation Bilinear interpolation

dycxbyaxyxv '''')','(

83



84



Documents

Chapter 5 Image Restoration