박 사 학 위 논 문

Doctoral Thesis

편미분방정식 기반의 영상 분할 방법과 복원 방법

PDE-based image processing for segmentation

and image restoration

한 주 영 (韓 周 寧 Hahn, Jooyoung)

수리과학과

Department of Mathematical Sciences

한 국 과 학 기 술 원

Korea Advanced Institute of Science and Technology

2008

편미분방정식 기반의 영상 분할 방법과

복원 방법

PDE-based image processing for segmentation

and image restoration

PDE-based image processing for segmentation

and image restoration

Advisor : Professor Lee, Chang-Ock

by

Hahn, Jooyoung

Department of Mathematical Sciences

Korea Advanced Institute of Science and Technology

A thesis submitted to the faculty of the Korea Advanced

Institute of Science and Technology in partial fullfillment of

the requirements for the degree of Doctor of Philosophy in the

Department of Mathematical Sciences

Daejeon, Korea

2007. 11. 29.

Approved by

Professor Lee, Chang-Ock

Major Advisor

편미분방정식 기반의 영상 분할 방법과

복원 방법

한 주 영

위 논문은 한국과학기술원 박사학위논문으로 학위논문심사

위원회에서 심사 통과하였음.

2007년 11월 29일

심사위원장 이 창 옥 (인)

심사위원 김 홍 오 (인)

심사위원 권 길 헌 (인)

심사위원 박 현 욱 (인)

심사위원 서 진 근 (인)

DMAS

20025321

한 주 영. Hahn, Jooyoung. PDE-based image processing for segmentation

and image restoration. 편미분방정식 기반의 영상 분할 방법과 복원 방

법. Department of Mathematical Sciences. 2008. 111p. Advisor Prof. Lee,

Chang-Ock. Text in English.

Abstract

We propose noble ideas and formulations based on nonlinear partial differential

equations (PDEs) in image segmentation and image restoration. Two algorithms

which have different perspectives on taking initial contours are proposed in image

segmentation. The first is to place initial contours arbitrarily for the purpose of cap-

turing multiple junctions and holes of objects in an image. The second is to place

those close to boundaries of objects for the purpose of fine segmentation. In image

restoration, we propose a nonlinear PDE for regularizing a tensor which contains

the first derivative information of an image such as strength of edges and a direction

of the gradient of the image. It improves the quality of results in many low level

topics in computer vision, which need the first derivative information of an image.

In the first segmentation algorithm, noble forces based on active contours models

are proposed for capturing objects in an image. The main purpose of segmentation

is to detect multiple junctions and holes of objects in an image. Contemplating the

common functionality of forces in previous active contours models, we propose the

geometric attraction-driven flow (GADF), the binary edge function, and the binary

balloon forces to detect objects in difficult cases such as varying illumination and

complex shapes. The orientation of GADF is orthogonally aligned with a boundary

of an object and two vectors across the boundary are the opposite direction. Since

GADF is obtained robust to changes of strength of edges, it prevents a leakage on

an weak edge in curve evolution. To reduce the interference from other forces, we

design the binary edge function using the property of orientation in GADF. We also

design the binary balloon forces based on the four-color theorem. Combining with

initial dual level set functions, the proposed model captures holes in objects and

multiple junctions from different colors. The result does not depend on positions of

initial contours.

In the second segmentation algorithm, we propose fine segmentation in order to

i

extract objects in an image without loss of detailed shapes. The proposed method

is well performed for the image which has simple background colors or simple object

colors. The GADF and edge-regions are combined to detect boundaries of objects

in a sub-pixel resolution. The main strategy to segment the boundaries is to con-

struct initial curves close to objects by using edge-regions and then to make a curve

evolution in GADF. Since the initial curves are close to objects regardless of shapes,

highly non-convex shapes are naturally detected and dependence on initial curves in

boundary-based segmentation algorithms is removed. Moreover, weak boundaries

are captured because the orientation of GADF is obtained robust to changes of

strength of edges.

According to the main purpose of segmentation which is fine extraction of objects

or measurement of sizes of objects, we propose a local region competition (LRC)

algorithm. This noble algorithm detects perceptible boundaries which can be used

to extract objects from an image without visual loss of detailed shapes. The LRC

and edge-regions make distinctive difference from the first segmentation algorithm.

We have successfully accomplished fine segmentation of objects from images taken

in a studio and aphids from images of soybean leaves.

In image restoration, we propose a nonlinear partial differential equation (PDE)

for regularizing a tensor which contains the first derivative information of an image

such as strength of edges and a direction of the gradient of the image. Unlike a

typical diffusivity matrix which consists of derivatives of a tensor data, we propose

a diffusivity matrix which consists of the tensor data itself, i.e., derivatives of an

image. This allows directional smoothing for the tensor along edges which are not in

the tensor but are in the image. That is, a tensor in the proposed PDE is diffused fast

along edges of an image but slowly across them. Since we have a regularized tensor

which properly represents the first derivative information of an image, the tensor

is useful to improve the quality of image denoising, image enhancement, corner

detection, ramp preserving denoising, image inpainting, and image magnification.

We also prove the uniqueness and existence of solution to the proposed PDE.

ii

Contents

Abstract i

Contents iv

List of Tables vi

List of Figures viii

1 Introduction 1

2 Image Segmentation 7

2.1 Motivation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7

2.2 Segmentation using GADF . . . . . . . . . . . . . . . . . . . . . . . 9

2.2.1 Overview: common terms in active contours . . . . . . . . . . 9

2.2.2 Geometric attraction-driven flow . . . . . . . . . . . . . . . . 15

2.2.3 Binary edge function . . . . . . . . . . . . . . . . . . . . . . . 24

2.2.4 Binary balloon force . . . . . . . . . . . . . . . . . . . . . . . 25

2.2.5 Examples and numerical aspects . . . . . . . . . . . . . . . . 30

2.3 Fine segmentation using edge-regions . . . . . . . . . . . . . . . . . . 38

2.3.1 Overview: algorithms . . . . . . . . . . . . . . . . . . . . . . 38

2.3.2 Step 1: Detection of edge-regions . . . . . . . . . . . . . . . . 40

2.3.3 Step 2: Construction of initial curves for segmentation . . . . 46

2.3.4 Step 3: Placement of curves on exact boundary . . . . . . . . 51

2.3.5 Step 4: Local region competitions for perceptible boundary . 53

2.3.6 Examples and numerical aspects . . . . . . . . . . . . . . . . 55

3 Image Restoration 62

3.1 Motivation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62

iv

3.2 A nonlinear PDE for regularizing a tensor . . . . . . . . . . . . . . . 65

3.2.1 Modeling of PDE . . . . . . . . . . . . . . . . . . . . . . . . . 65

3.2.2 Quality of the nonlinear structure tensor . . . . . . . . . . . . 68

3.2.3 Different types of PDEs . . . . . . . . . . . . . . . . . . . . . 73

3.3 Existence and uniqueness . . . . . . . . . . . . . . . . . . . . . . . . 75

3.4 Applications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 84

3.4.1 Corner detection . . . . . . . . . . . . . . . . . . . . . . . . . 84

3.4.2 Image denoising and enhancement . . . . . . . . . . . . . . . 86

3.4.3 Ramp preserving denoising . . . . . . . . . . . . . . . . . . . 93

3.4.4 Image inpainting . . . . . . . . . . . . . . . . . . . . . . . . . 96

3.4.5 Image magnification . . . . . . . . . . . . . . . . . . . . . . . 99

4 Conclusion 102

References 105

v

List of Tables

2.1 General problems in active contours for image segmentation . . . . . 8

2.2 Comparison of active contours based on (2.8). Fs is to control smooth-

ness of contours, Fb is to force contours to move from far distance

toward the boundary of objects, Fa is to attract contours much closer

to the boundary. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14

2.3 When the proposed active contours model (2.9) is used, we check the

success of capturing the object for different Fb and different positions

of zero level set Γφ0 of the level set function φ0. Note that, even

though Γφ0 = Γ−φ0 , they give different results. Four different posi-

tions of the initial contour Γφ0 and the regions Ω1 and Ω2 are shown

in Figure 2.3. Notice that the results of using Fb with φ0 and −Fbwith −φ0 in the Cases III and IV are exactly same. We call the pairof level set functions φ0 and −φ0 as initial dual level set functions. . 26

2.4 It shows the robustness of the proposed algorithm to different noise

levels. The Gaussian white noise is added with the zero mean and

the different values of the standard deviation σ from 10 to 100. The

accuracy is computed by using the relative length in (2.24). The

images on the right show different noise levels, from the top left to

the bottom right, σ = 20, σ = 50, σ = 70, and σ = 100. . . . . . . . 34

vi

3.1 (a) is a clean image Ic. In (b), we add the Gaussian white noise with

zero mean and the standard deviation 70 (SNR ' 7.62). (c), (d), and(e) are obtained by different PDEs for image denoising (3.13), (3.14),

and (3.15) at T1 = 30, respectively (see Section 3.2.3). From top to

bottom, different end time T2 for regularizing a tensor is used as 1,

5, and 10. SNR and relative H1 norm error are computed by (3.9)

and (3.10), respectively. Note that denoised images in (e) from (3.15)

which uses our regularized tensor (3.7) preserve geometric features

such as edges and corners in the original image (a) and have steady

and high SNR with various end time T2. . . . . . . . . . . . . . . . . 69

vii

List of Figures

1.1 The left image is a digital image. The graph in the middle is the

intensity profile of the left image. It assumes that an image is a

positive real-valued function defined on a rectangular domain even

though a digital image is represented by integers from 0 as black to

255 as white. The right image shows the symmetric and periodic

extension of the left image. . . . . . . . . . . . . . . . . . . . . . . . 2

2.1 We compare the orientation of GADF with the orientation of the

GVF and the gradient of edge function ∇g. Note that ∇g ‖ ∇fwhere f is the edge map. If two vectors have opposite direction, they

are highlighted with the yellow color in (d), (e), and (f). Clearly, the

GADF in (a) has better information than the GVF and the gradient

of edge function along the boundary of object. . . . . . . . . . . . . 19

2.2 The GADF (2.13) for an image as a real-valued function I: [0, 1] ⊂R → R+. From the top, the graphs are the image I, I ′, and I ′′.The point x0 is the edge (2.16). The sign at the bottom is obtained

by (2.17) and the arrows represent the GADF. . . . . . . . . . . . . 21

2.3 (a) is the original image and the black object is placed in the middle.

(b) shows the regions in (2.22) after the binary edge function is ob-

tained. We denote Ω1 and Ω2 as connected components of Ωb which

is the support of the binary edge function. (c) is different initial con-

tours Γφ0 . Note that the initial level set function φ0 is positive inside

the contour and negative outside the contour. . . . . . . . . . . . . . 25

viii

2.4 From top to bottom, the evolving contours of the proposed model (2.9)

are shown with different positions of the initial contour. We use dual

level set functions, φ0 and −φ0, as the initial condition and the binaryballoon force as the Case III in Table 2.3. The level set function φ0 is

positive inside the contour and negative outside the contour. The red

and blue contour are evolved from φ0 and −φ0, respectively. Noticethat, by using initial dual level set functions and the proper binary

balloon force, the proposed model has the capability of capturing the

object regardless of the position of initial contours. . . . . . . . . . 27

2.5 (a) is the original image and the object in the middle has holes and

multiple junctions. The gray region in (b) is Ωa and four colors are

labeled on connected components of Ωb in (2.22) based on the four-

color theorem. (c) and (d) are the profiles of F 1b and F2b ; white, gray,

and black represent 1, 0, and −1, respectively. The green contoursin (f) are initial contours. The contours in (g) are the result of the

proposed model (2.9) from initial dual level set functions with the

binary balloon force F 1b . The contours in (h) are obtained in the

same way from F 2b . Combining two contours in (g) and (h), the final

result in (e) is obtained. The evolving contours from (f) to (g) and

(h) are shown in the third and the fourth row, respectively . . . . . 29

2.6 The contours in (b) and (d) are the results of the proposed model (2.9)

from the initial contours in (a) and (c), respectively. The GADF

captures the weak edge which is the left part of the rectangle frame.

The RAGS in (f) also capture the weak edge when the region map

gives the correct information. In (e) and (f), we use two different

region maps to obtain the result of the RAGS. The contour from the

GAC and the GVF passes by the weak edge. Note that we capture

the whole frame in (d) by using the general type of initial contours in

(c). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31

ix

2.7 The initial contour in each method is placed at the boundary of image.

The moving contours are shown from top to bottom. The illumination

is changed on the object and its background. The GADF clearly

captures the object. Note that the region map in the RAGS is almost

same as the object. The ACWE in (d) may capture the object by

manipulating four parameters in the formulation (2.6), however, it

does not work with λ1 = λ2 = 1, α = 0.01, and η = 0. The simplified

version of the GAR [48] in (e) does not capture the rectangle because

the Gaussian distribution is not adjustable to represent the varying

illumination. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32

2.8 Multiple junctions and holes are captured by the proposed model.

Note that the varying illumination on each hole is caused by the

shadow. It makes the same difficulty to capture the object as in

Figure 2.7. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33

2.9 The images are taken in the photo studio. The objects are commercial

products, the first row is a part of DVD player and the others are

parts of a component in a machine. Even though the objects are

taken on the simple background, there are well-known difficulties in

image segmentation: the weak edge and complexity of shapes such as

holes and multiple junctions. Note that weak edges are shown in the

right side of DVD in the first row and the bottom of the object in the

last row. The proposed model clearly captures boundaries of objects. 36

2.10 We use some of images from [30,31]. The proposed model captures the

object in each image. The holes in the object and multiple junctions

from different colors are detected, but some of thin branch in the first

row and the left ventral fin of the bream in the third row are not

clearly segmented. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37

x

2.11 The vector field in the left image is the GADF (2.13) and the blue

curve is the exact boundary obtained from the segmentation algo-

rithm in the previous section. The significant difference between the

exact boundaries and the perceptible boundaries is show on the right

diagram. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38

2.12 In (a), an ideal case of normalized vectors ~w/|~w| around x0 whichrepresents an edge are shown. (b) is a small part of Figure 2.23. The

vector in (c) is the normalized vector field ~w/|~w| and the black regionin (c) represents the candidates of edge-regions. . . . . . . . . . . . 42

2.13 Bad candidates and edge-regions: (a) is an original image. The black

region in (b) represents candidates of edge-regions and the candidates

in yellow areas are bad candidates which are not parts of boundaries

of objects obviously. The black regions in (c) are the edge-regions. . 44

2.14 Procedures of solving (2.34): (a) is a small part of Figure 2.23. The

black regions in (b) are edge-regions and the curves Γψ(·,0) in (b) are

the initial condition for (2.34). The curves Γψ(·,T1) in (c) are a result

of (2.34), which connects edge-regions along boundaries of objects.

The curve Γψ̃

in (d) is obtained from the curves in (c) and will be

used as an initial curve for the segmentation process. . . . . . . . . 47

2.15 Eigenvectors vλ and evolving curves ψ(x, t): The concave parts of

curves will stay because of the restriction in (2.36). The curves in (a)

are an initial condition for (2.34) and the curves in (b) are obtained

at time T1. Note that the image is a small part of Figure 2.23. . . . 48

xi

2.16 The effect of the force Fs(x) in the result of (2.34): a result without

using Fs(x) has excessive connections. The curves in (a) are the initial

condition Γψ(·,0) for (2.34). The profile in (b) is Fs(x). The curves

Γψ(·,T1) in (c) are a result in (2.34) without using Fs(x). The curves

Γψ̃

in (d) from (c) are bad initial curves for the segmentation process

since they are far from the object. The curves Γψ(·,T1) in (e) are a

result in (2.34) with Fs(x). The curves Γψ̃ in (f) from (e) are good

initial curves for the segmentation process. Note that the image is a

small part of Figure 2.23. . . . . . . . . . . . . . . . . . . . . . . . . 50

2.17 Initial curves and the result of (2.37): While the contours of simply

connected regions are disappeared, the outer contour of multiply con-

nected regions stays at exact boundaries of objects. The vectors are

GADF. A result Γψ(·,T1) from (2.34) is in (a). The image (b) shows

where simply connected regions and a multiply connected region are.

The curves Γψ̃

in (c) are initial curves Γφ(·,0) for (2.37). The curve

in (d) is a result of segmentation from (2.37). Note that the image is

a small part of Figure 2.23. . . . . . . . . . . . . . . . . . . . . . . . 52

2.18 Conceptual diagram for understanding the force F̂s in (2.38). The

curve Γϕ(·,t) is going to move inward. . . . . . . . . . . . . . . . . . . 55

2.19 The perceptible boundary, the exact boundary, and the extracted

objects: The image in (a) is a small part of Figure 2.23. In (b), the

red curve is the perceptible boundary and the blue curve is the exact

boundary. The extracted object from the perceptible boundary is

with white background in (c). The extracted object from the exact

boundary is with white background in (d). See visual loss of the

object in (d), which can be hardly seen in (c). . . . . . . . . . . . . . 56

xii

2.20 A procedure of a fine segmentation algorithm using GADF and edge-

regions: (a) is an original image. The black regions in (b) are edge-

regions. The curves in (c) are a result of (2.34). In (d), initial curves

for a segmentation process are shown. The curves in (e) are percep-

tible boundaries. The image in (f) is an extracted object on white

background. The size of image is 940 by 544. . . . . . . . . . . . . . 57

2.21 An example for 3D VR content: The yellow part is highly non-convex.

The size of image is 468 by 576. . . . . . . . . . . . . . . . . . . . . 58

2.22 An example for 3D VR content: As a smooth lighting condition is

used, the original color of shiny surface is seriously changed because

of total reflection. It generates weak edges near boundaries of the

object. The size of image is 288 by 288. . . . . . . . . . . . . . . . . 58

2.23 An example for 3D VR content: It has a repeated non-convex shape

and weak edges changed smoothly from strong edges due to a reflec-

tion of light. The size of image is 1056 by 496. . . . . . . . . . . . . 59

2.24 Segmenting aphids in a soybean leaf: Original image and segmenta-

tion of aphids. The size of each image is 640 by 480. From top to

down, it is getting old leaves. Since the color is changed depending

on the age of leave, the color extraction approach does not work. . . 61

3.1 (a) is an one-dimensional image data I(x). The point x = e is an

edge in (a). (b) is u(x, 0) =(dIdx(x)

)2. (c) is

∣∣∂u∂x(x, 0)

∣∣2. . . . . . . . . 653.2 (a) is an one-dimensional noisy image data I(x) (green) and an orig-

inal data (red). (b) is an initial data of (3.6) (green) and the square

of derivative of the original data (red). (c) is regularized derivatives

at T2 = 600 using different PDEs. The green curve is a result of the

proposed PDE (3.6) and the blue curve is a result of (3.5). . . . . . 67

xiii

3.3 (a) is a given image and the red curve in (b) is the exact location of

edges. (c) is the vector field V in (3.11) obtained by the regularizedtensor of the proposed PDE (3.7) at T2 = 10. Vectors on the set Rin (3.12) are highlighted in yellow. (d) and (e) are magnified images

from both the red curve in (b) and vectors in (c) on green square

regions. Note that vectors in highlighted in yellow are placed in pixels

close to edges. They also point to edges and are aligned orthogonally

to edges. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 71

3.4 A restricted vector field V|R is obtained by (3.11) and (3.12) from atensor. Vectors in (a) are V|R from a tensor ∇Ĩ∇ĨT, where a regular-ized image Ĩ is obtained by the PM model (3.13). Vectors in (b), (c),

(d), and (e) are V|R from the regularized tensors of (3.16), (3.17), (3.18)and (3.19), respectively. From top to bottom, end time T1 for regu-

larizing an image and T2 for regularizing a tensor are taken as same

values, 11, 33, and 55. Apparently, vectors from our regularized ten-

sor (3.19) show the best result for preserving derivative information

near edges in an image as time evolves. . . . . . . . . . . . . . . . . . 72

3.5 (a) is an original image. In (b), we add the Gaussian white noise with

zero mean and the standard deviation 50 (SNR ' 14.49). From (c) to(f), they are profiles of minimum eigenvalues from different regular-

ized tensors in (3.16), (3.17), (3.18), and (3.19) at end time T2 = 70,

respectively. Images from (c-1) to (f-1) are magnified from a part on

the top right rectangle from (c) to (f) and we plot intensity graphs

of the minimum eigenvalue with the part of the image. The profile

(f-1) shows the best result which has four sharp peaks at corners and

flatter shape on homogeneous regions in the image. . . . . . . . . . . 85

xiv

3.6 The comparison of classical denoising algorithms. From the left on

top, we have the original image which is corrupted by granular noise in

the negative film, the image denoised by the proposed algorithm (3.32),

the image denoised by the median filter, and the last image denoised

by the Gaussian filter. The figures on the bottom represent intensity

plots of the corresponding gray image. We can see how the initial im-

age is corrupted. The median filter shows a typical stair case problem

and the Gaussian filter deteriorates detailed shapes. The proposed al-

gorithm gives the best result. . . . . . . . . . . . . . . . . . . . . . . 87

3.7 (a) is an original clean image. In (b), we add the Gaussian white

noise with zero mean and the standard deviation 50 (SNR ' 11.97).We use end time T2 = 5 for obtaining a diffusivity matrix in (3.33).

The result (c) is a combination of the Perona-Malik model for a color

image with a fidelity term and a shock filter. (d) is the result of

proposed method (3.33) which preserves corners and edges very well.

In the second and the third row, we magnify two parts in the first row. 88

3.8 (a) is an original image which has jpeg artifacts. In (b), we add

the Gaussian white noise with zero mean and the standard deviation

10. We use the end time T2 = 10 for obtaining a diffusivity matrix

in (3.33). (c) and (d) are obtained in the same way as in Figure 3.7-

(c) and 3.7-(d), respectively. The result in (d) from the proposed

method preserves significant features around the frame of glasses. In

the second row, we magnify a region around the right eye in the first

row. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 89

3.9 The corrupted image by bad weather and inadequate ISO in a digital

camera. It is enhanced in Figure 3.10. . . . . . . . . . . . . . . . . . 90

3.10 The enhanced image obtained by (3.33) from the corrupted image in

Figure 3.9. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 91

xv

3.11 Images in (a) are an original clean image at the top row and an image

at the bottom row with the Gaussian white noise with zero mean

and the standard deviation 20 (SNR ' 14.56). (b) is a result of theproposed model (3.34) at T1 = 17. We use different end time T2 = 1

at the top row and T2 = 5 at the bottom row. The result preserves

ramp structure of the original image. . . . . . . . . . . . . . . . . . . 93

3.12 (a) is same images in Figure 3.11-(b). (c) is a vector field obtained

by (3.11) and (3.12) from a tensor in (3.34). The position of yellow

vector field indicates where the maximum eigenvalue of a tensor has

a local maximum. The noisy data in Figure 3.11-(a) is diffused slowly

across yellow regions in the proposed model (3.34) and ramp structure

is preserved. In (b), we show both the left of (a) and the right of (c). 94

3.13 We magnify images in Figure 3.12-(c) in order to see the direction of

vector field. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 95

3.14 We use an image from the Berkeley Image Database [31]. (a) is an

original image. The green curve in (c) is the region D contains the

damaged image and the image (d) shows the initial condition in (3.35).

The image (e) is the result at T1 = 500. In (b), we put the inpainted

image (e) into the original image to see how it looks naturally. . . . 97

3.15 The top is an original image, the middle is an initial image in (3.35),

and the bottom is recovered image. The brand name is clearly deleted. 98

3.16 (a) is original image. (b) is the magnified image from (a) in Section 3.4.5.100

xvi