Content-Based Image Retrieval Using Multiresolution Color and Texture Features

Young Deok Chun, Nam Chul Kim, Member, IEEE, and Ick Hoon Jang, Member, IEEE

IEEE TRANSACTIONS ON MULTIMEDIA,OCTOBER 2008

Outline

Introduction Conventional features Proposed image retrieval method Experimental results Conclusion

Introduction

Typical CBIR Extract features related to visual content from a

query image Compute similarity between the features of the

query image and target images in DB Target images are next retrieved which are most

similar to the query image Extraction of good features is one of the

important tasks in CBIR Shape▪ Describes the contours of objects in an image▪ Usually extracted from segmenting the image into

meaningful regions or objects

Introduction

Color▪ Most widely used visual features ▪ Invariant to image size and orientation

Texture▪ Visual feature that refers to innate surface properties of an object ▪ Relationship to the surrounding environment

A feature extracted from an image is generally represented as a vector of finite dimension Feature vector dimension is one of the most important

factors that determine▪ The amount of storage space for the vector▪ Retrieval accuracy▪ Retrieval time (or computational complexity)

Conventional features

Explain the conventional features which are used in the proposed retrieval method color feature▪ color autocorrelogram

texture features▪ BDIP ▪ BVLC

Color Autocorrelogram

Color autocorrelogram Captures the spatial correlation between identical colors

only Probability of finding a pixel p’ of the identical color at a

distance k from a given pixel p of the lth color

measure the distance between pixels

As k varies, spatial correlation between identical colors in an image can be obtained in various resolutions

IlIpforlIpandkppIplk ''' |Pr

1,.....1,0 LI

''' ,max yyxxpp

BDIP

Block Difference of Inverse Probabilities Texture feature that effectively extracts

edges and valleys

Object boundaries are extracted well , and the intensity variation in dark regions is emphasized

yxI

yxIyxIB

lkl

kl

kl

Byx

Byx Byxklk

,max

,,max1

,

, ,

Block size (k+1)*(k+1)

Intensity of pixel(x,y)

Representative (maximum) of intensity variation in a block

Representative value in a block

Original image BDIP operator

BVLC

Block Variation of Local Correlation coefficients Represents the variation of block-based local

correlation coefficients according to four orientations

Local correlation

BVLC

0,,,0,,0,0,4 kkkkO

klkll k

Ok

k

Ok

k

,min,max44

kll

Byx kllyxklk

kl

kykxIyxIB

kl

,

,,1

,standard deviation ofthe lth block shifted by k

Local covariance normalized by local variance

Difference between the max and min values of block-based local correlation coefficients (four orientations)

Proposed image retrieval method

Overall structure

A texture feature vector ft ofdimension Nt with BDIP and BVLC moments extracted fromV

nmW ,

Each component image is waveletdecomposed into a wavelet image

4-band wavelet decomposition

configuration of 2-level wavelet decomposed images

A color feature vector fc of dimension Nc is then formed with color autocorrelograms extracted from the and

HnmW ,

SnmW ,

Color Feature Extraction

Color Feature Extraction

The and are first quantized into and LL band▪ uniformly quantized

other subbands ▪ non-uniformly quantized by the generalized Lloyd

algorithm Given the total number of quantization levels L of

all subbands, Li is allocated to the ith subband

HnmW ,

SnmW ,

HnmQ ,

SnmQ ,

1

0

K

iiLL KK

j j

ii K

LL 1

1

0

2

2

22

2 log2

1loglog

Color Feature Extraction

Numbers of quantization levels for all subbands are decided

Color autocorrelogram is extracted To reduce computational complexity, we modify the color

autocorrelogram

Cnm

Cnm

Cnm

Cnmk

nmQlQpforlQpand

pNpandkppQplC

,,,'

2''

,'

, Pr,

ZnHHLHHLLLm

LlSHC nm

,...,1,,,,

,1,...,1,0,, ,

the set of pixels having the lth color

Color Feature Extraction

The color feature vector fc is finally formed with the color autocorrelogram

color autocorrelogram probability of finding a pixel p’ of the lth color among

the two causal neighboring pixels at a distance k from a given pixel p of the lth color in the

lCf knmc ,,

ZnHHLHHLLLm

LlSHC nm

,...,1,,,,

,1,...,1,0,, ,

CnmQ ,

Texture Feature Extraction

Texture Feature Extraction

is first divided into nonoverlapping blocks of a given size, where the BDIP and BVLC are computed

BDIP the denominator in may yield negative BDIP values in

wavelet domain, which leads to invalid measurement of intensity variation

yxW Vnm ,,

yxW

yxWyxWB

lVnLLByx

Byx

Vnm

VnmByxk

lk

kl

kl

kl

,max

,,max1

,,

, ,,,

ZnHHLHHLLLm ,...,1,,,, Pixel values are nonnegative in LL band

Texture Feature Extraction BVLC

Reduce computational complexity▪ Local covariance▪ Mean absolute difference of pixels between two

blocks

▪ Local variance ▪ Mean absolute difference of four end pixels in the

block

2

,,1

,, ,,

,kll

Byx yxVnm

Vnmk

lknm

kl

kykxWyxWB

kl

kykxWkyxWykxWyxW

kykxWykxWkyxWyxW

Vnm

Vnm

Vnm

Vnm

Vnm

Vnm

Vnm

Vnm

l,,,,

,,,,

4

1

,,,,

,,,,

Texture Feature Extraction Modified BVLC

Difference between the max and min values of the local correlation coefficients according to two orientations

klkll k

Ok

k

Ok

knm

,min,max

22,

kkO ,0,0,2

Texture Feature Extraction The first and second moments of the BDIP

and BVLC for each subband are extracted

The texture feature vector

2,,,2

,,

2

,,,2

,,

knm

knm

knm

knm

knm

knm

knm

knm

knm

knm

l

l

l

l

knm

knm

knm

knmTf ,,,, ,,,

ZnHHLHHLLLm ,...,1,,,,

Feature Vector Combination and Similarity Measurement

Feature Vector Combination After color and texture feature vectors are extracted,

the retrieval system combines these feature vectors

Each of the color and texture feature components is normalized by its dimension and standard deviation reducing the effect of different feature vector dimensions and

component variances in the similarity computation

knm

knm

knm

knmTf ,,,, ,,,

lSlHf knm

knmc ,,, ,,

ZnHHLHHLLLm ,...,1,,,,

TT

T

CC

C

N

f

N

ff

,

Similarity Measurement

Use generalized Minkowski-form distance of metric order one

Feature dimension Color feature

▪ NC : determined as the total number of quantization levels L

Texture feature▪ NT : 16Z

N= NC + NT The number of additions for a query image in the

similarity measurement of the retrieval is given as

N

i

tqtq ififffD1

,

120 NK

Similarity Measurement

Huge Database Progressively implement▪ reduce the computational complexity in CBIR

Set of candidate images is selected by feature matching at the lowest level

Progressive refinement is performed as the level increases

Progressive retrieval is composed of Z+1 steps▪ First step▪ the color feature vector fC and the texture feature vector fT are combined for

m=LL and n=1

▪ Second step▪ m={LL,HL,LH,HH} and n=1

▪ Z+1 step▪ m={LL,HL,LH,HH} and n={1,….,Z}

Similarity Measurement

At each step (jth step) query ▪ the combined feature vector fj

q of dimension Nj

Target▪ the combined feature vector fj

t of the same dimension

▪ for each of kj-1 target images

kj target images with the best similarity are retrieved Total number of additions for a query in the similarity

measurement of the progressive retrieval

where kj is determined to decrease and the Nj increases as the retrieval step j increases

Z

jjj NK

01 12

EXPERIMENTAL RESULTS

Image Database The Corel DB▪ 990 RGB color images▪ 192 x 128 pixels▪ 11 classes, each of 90 images

VisTex DB ▪ 1200 RGB color images▪ 128 x 128 pixls▪ 75 classes, each of 16 images

MPEG-7 common color dataset (CCD) ▪ 5420 color images▪ 332 ground truth sets (GTS)

where the number of images in each GTS varies

EXPERIMENTAL RESULTS

Corel MR DB▪ directly from a third of all the images for each class in the

Corel DB▪ Ratio of (1.5:1)▪ Ratio of (2:1)

VisTex MR DB▪ Ratio of (1:1),(1.5:1),(1.75:1),(2:1)

MPEG-7 CCD MR DB▪ Ratio of (1:1),(1.5:1),(2:1)

The sizes and numbers of classes of the three derived DBs are the same as those of the original DBs

Contain images of various resolutions

EXPERIMENTAL RESULTS

Performance Measures For a query q

A(q) : a set of retrieved images in a DB B(q) : images relevant to the query q

Precision

Recall

ANMRR (average normalized modified retrieval rank)▪ measure of retrieval accuracy used in almost all of the MPEG-7 color

core experiments▪ The ANMRR gives just one value for a DB▪ Lower ANMRR value means more accurate retrieval performance

qA

qBqAqP

qB

qBqAqR

EXPERIMENTAL RESULTS

Specifications of Retrieval Methods wavelet decomposition level was chosen as Z=2 total number of quantization levels L for each of the wavelet

decomposed H and S component images was also chosen as L=30▪ {Lm,1} = {8,4,4,4} , {Lm,2} = {4,2,2,2}

Dimensions of feature vector▪ NC = 60

▪ NT = 32

Z=2 , Proposed progressive retrieval has 3 steps▪ At the step of j=1,2,3 , the total number of quantization levels for

each of the wavelet decomposed H and S component images is given as L = 8,20, and 30

feature vector dimensions▪ (N1,N2,N3) =(20,56,92)

EXPERIMENTAL RESULTS

The proposed method with nonprogressive scheme and that with progressive scheme

(a) Corel DB (b) VisTex DB.

former yields the average precision loss of 1.5% and that of 1.1% over the latter for Corel DB and for VisTex DB

Proposed retrieval method with progressive scheme is about 1.2 times faster than nonprogressive scheme

EXPERIMENTAL RESULTS

Precision versus recall of the proposed method with progressive scheme according to each step

(a) Corel DB (b) VisTex DB.

EXPERIMENTAL RESULTS

Single features and the proposed progressive retrieval method

Combination of color and texture features and the proposed progressive retrieval method

EXPERIMENTAL RESULTS

ANMRR of the retrieval methods

The proposed method almost always yields better performance in precision versus recall and in ANMRR over the other methods for the six test DBs

EXPERIMENTAL RESULTS

Retrieval ranks of the relevant images for the query image

Query imageResolution is identical with the query image

The proposed method is more effective for multiresolution image DBs

Conclusion

The feature vector is scalable according to the decomposition level Z in the wavelet transform domain It was found in some experiments that the retrieval

accuracies of Z>2 are slightly better than those of Z=2

Experimental results for six test DBs showed that the proposed method yielded higher retrieval accuracy than the other conventional methods

It was all the more so for multiresolution image Databases