Talal_Imran

8/2/2019 Talal_Imran

1/7

Image Classification and Retrieval using Correlation

Imran Ahmad

School of Computer ScienceUniversity of Windsor

Windsor, ON N9B 3P4 - Canada

[email protected]

Muhammad Talal Ibrahim

Dept. of Computer ScienceCOMSATS Institute of Information Technology

Islamabad - Pakistan

[email protected]

Abstract

Image retrieval methods aim to retrieve relevant images

from an image database that are similar to the query image.

The ability to effectively retrieve non-alphanumeric data is

a complex issue. The problem becomes even more difficult

due to the high dimension of the variable space associatedwith the images. Image classification is a very active and

promising research domain in the area of image manage-

ment and retrieval. In this paper, we propose a new image

classification and retrieval scheme that automatically se-

lects the discriminating features. Our method consists of

two phases: (i) classification of images on the basis of max-

imum cross correlation and (ii) retrieval of images from the

database against a given query image. The proposed re-

trieval algorithm recursively searches similar images on the

basis of their correlation against a given query image from

a set of registered images in the database. The algorithm

is very efficient, provided that the mean images of all of the

classes are computed and available in advance. The pro-

posed method classifies the images on the basis of maximumcorrelation so that the images with more similarities and,

hence, exhibiting maximum correlation with each other are

grouped in the same class and, are retrieved accordingly.

1. Introduction

With advances in computing and digital imaging tech-

nologies, the number of images is increasing rapidly, thus,

making it necessary to provide techniques for efficient man-

agement and retrieval of stored images. Image classifica-

tion and retrieval is also a major issue in the areas of pat-

tern recognition, robotics and artificial intelligence. Exam-ples of systems requiring classification include but are not

Author would liketo acknowledge partialsupport provided by the Nat-

ural Sciences and Engineering Research Council (NSERC) of Canada and

Higher Education Commission (HEC) of Pakistan for completion of part

of this work.

limited to visual tracking [7], image registration [6], and

content-based image retrieval [2].

Essentially, there are two main image retrieval tech-

niques: (i) text-based retrieval and (ii) content-based re-

trieval. All of the earlier image retrieval techniques were

text based and involved annotations of salient image fea-

tures or association of keywords with the image contents.Retrieval was done by issuing a text-based query and only

on the basis of successful match of the keywords or the

annotations. Even though text-based approaches are able

to capture the high level abstractions and concepts asso-

ciated with the image contents, it is believed that due to

manual procedures for annotations of image contents, such

approaches are fraught with difficulties. Manual proce-

dures are not only time consuming but are highly subjec-

tive. Moreover, some of the visual aspects of images are

inherently difficult to describe while others could equally

be described in many different ways [4]. Given the huge

amount of image data that exists now or will be collected

in future, manual approaches are clearly inadequate. In

order to eliminate such problems and to make image re-

trieval more efficient, there has been a great emphases on

retrieval techniques that are based on automatic extraction

of visual features and mathematical attributes from the im-

age contents. Such type of retrieval is generally known as

the Content-based Image Retrieval (CBIR) and has been

an active area of research for many years. Even though

CBIR approaches (see [12] for a partial list of such tech-

niques) can capture directly computable low level features

from the images, efficient and precise image retrieval still

remains to be an open problem. Despite development of

many commercial and/or academic/research CBIR systems

over the years, researchers are still actively working to find

a more comprehensive solution with an objective to provideimproved precision and recall.

Auto correlation and cross correlation are standard statis-

tical techniques and are used in the areas of image process-

ing and pattern recognition to estimate the degree to which

two given patterns are correlated. These techniques have

Proceedings of the 3rd Canadian Conference on Computer and Robot Vision (CRV06)

0-7695-2542-3/06 $20.00 2006IEEE


2/7

been successfully applied to estimate the motion of moving

objects [10], relevance feedback in CBIR systems [8] and

image registration, etc. [10].

In this paper, we propose an image classification and re-

trieval scheme that is based on the concept of maximum

cross correlation between images with promising results. In

the proposed approach, we first classify images on the ba-

sis of maximum cross correlation with each other and storethem accordingly whereas for the retrieval of an image, this

classification is used.

The rest of the paper is organized as follows: in Section

2, a brief review of few important existing techniques and

those related to our approach is presented. In Section 3,

the proposed system is discussed whereas some of the ex-

perimental results are presented and discussed in Section 4.

Finally, Section 5 provides some concluding remarks.

2. Related Work

Since 1990s, there has been a considerable progress in

the area of content-based image retrieval. In order to re-trieve and browse image data on the basis of their contents

and pictorial queries, many content-based image retrieval

systems have been proposed [1, 3, 4, 5, 15, 11, 13]. A sur-

vey of some of the important techniques is given in [12].

Such systems are essentially based on low level image fea-

tures that are directly computed from the image contents.

Some of the most commonly used features are the color, the

shape, the texture and the spatial locations and distributions

of the image objects [1, 4, 13].

Image retrieval is merely not restricted to search and rep-

resentation only. New issues are continuously introduced by

the subjectivity of the human perception and need to be ad-

dressed to provide satisfactory retrieval performance. Sys-

tems with relevance feedback are an attempt to address such

issues and to improve the quality of the retrieval. Such sys-

tems allow interaction with the user, who in turn, is respon-

sible for determining the quality of match and retrieval [9].

A new perspective in image retrieval method involving

a combination of factor analysis with relevance feedback

method is introduced in [9] whereas a statistical correlation

model for the retrieval of relevant images is presented in [8].

In this model, an estimate of the correlation between two

images based on the number of search sessions in which im-

ages have been marked relevant is calculated. Since in the

process of relevance feedback, main emphasis is to improve

the retrieval accuracy, several passes are made through the

database during the retrieval process and a correlation is dy-namically calculated by interacting with the user. However,

establishing a correlation dynamically is not only a time

consuming process but it also makes it difficult to incor-

porate positive and negative examples in query and/or the

similarity refinement process.

3. Proposed System

In Statistics terminology, correlation is a measure of the

relation between two or more variables. As mentioned ear-

lier, cross correlation and normalized correlation are stan-

dard statistical methods that have been successfully used

in various image related applications. Given the two same

size vectors or matrices, two-dimensional cross correlationbetween them can be calculated using the 2-D Correlation.

Assuming two matrices A and B of the same dimension

(m,n), the 2-D discrete normalized correlation can be com-puted as:

r =

m

n

(Amn A)(Bmn B)(m

n

(Amn A)2)(m

n

(Bmn B)2)

where A is the mean of the values of matrix A and B is

the mean of the values of matrix B and are given as:

A =1

mn

m1i=0

n1j=0

A(i, j)

and

B =1

mn

m1i=0

n1j=0

B(i, j)

The calculated correlation coefficients can range from -1

to +1 such that a value of -1 represents no correlation be-

tween the matching entities whereas a value of +1 repre-

sents a perfect positive correlation or a perfect match. Any

value in between is an indication of the degree of correla-

tion, and depending on application, can be used to make a

proper selection/retrieval decision.

Our proposed system has two phases: (i) classification or

clustering and (ii) retrieval. In the classification phase, im-

ages are classified on the basis of their maximum cross cor-

relation with each other according to the expression given

above and a collection of mean images corresponding to

each class is maintained. During the retrieval process, cross

correlation between the query image and the mean images is

determined and the class with maximum correlation satisfy-

ing the threshold criteria provided by the user is returned to

the user. A block diagram of the proposed system is given

in Figure 1. It is important to note that before the classifi-

cation process begins, all of the images are normalized to

make them all of same size.

Let the image database contains n discrete images andthat represents a user-defined threshold - minimum ac-

ceptable correlation between the query and the database im-

ages. The proposed recursive image classification algorithm

is then carried out by using one of the following three clas-

sification schemes:


0-7695-2542-3/06 $20.00 2006IEEE


3/7

New Image

CorrelationDetermination

Calculated correlationinformation of classified

images

ImageDatabase

Image Classification

Query Image

CorrelationDetermination

Image Retrieval

Retrieve class withmaximum correlation

ImageDisplay

Figure 1. System Architecture.

Linear classification: In this approach, images are

classified by making several passes over the database.

In the classification process, the first image from the

database is taken and its cross correlation with rest of

the images in the database is calculated. As a result, an

image with maximum correlation with the first one is

picked up and the two jointly form a new class. Now

we have two classes: one containing only two images

as discussed above and another one containing remain-

ing (n 2) images. We then take the first image fromthe class containing (n 2) images and calculate itscorrelation with rest of the images in its own class and

the class containing only two images. If its correla-

tion is maximum with the first class then it is moved

to the first class otherwise we pick the image from the

second class with which it has maximum cross correla-

tion and make a separate new class which will contain

only these two images. The same process is repeated

over and over until all of the images in the initial larger

class are classified. This scheme is very time consum-

ing with all worst, average and best case classification

time complexities ofn2 and results in maximum num-

ber of classes among the three classification methods

used in our experiments.

Selective classification: In this classification scheme,

we initially start with only two classes such that eachcontains only a single randomly picked image from the

database. The rest of the images are then classified on

the basis of their maximum correlation with the two

initial classes. Same process is repeated recursively

within each of the newly formed class and stops only

when it fails to satisfy the given threshold criteria,

defined at the beginning of the classification process. It

is important to note that the threshold can be the value

of the correlation coefficient such that 0 1 orsome number of images in a class. Result of the clas-

sification process is a binary tree in which leaf nodes

represent classes in which the images are maximally

cross correlated with other images in the same classand represent similar images.

Figure 2. Classes obtained using selectiveclassification approach.

Auto classification: In this scheme, we start by se-

lecting an image from the database and calculating its

correlation with rest of the images in the database and

choose the one that has the minimum correlation with

the first image. In this way, two initial classes are

formed that are totally independent of each other andhave minimum correlation between them. Now one by

one, we pick each of the remaining (n2) images andmake them part of one of the two newly formed classes

on the basis of their cross correlation. Same process is

repeated within each class as long as each class satis-


0-7695-2542-3/06 $20.00 2006IEEE


4/7

fies the criteria provided by the threshold parameter .

For retrieval against a given query image, the query im-

age is compared with the mean of each of the classes

and a correlation between the two is calculated. Im-

ages in a class having maximum cross correlation with

the query image are retrieved as the possible similar

images to the query image with a similarity measure

or the quality of match given by the difference in themagnitude of the cross relation of the query image and

that of the images in the retrieved class.

Figure 3. Classes obtained using auto classi-

fication approach.

The classification process in our approach is an expen-

sive process. Even though all of the three classificationschemes mentioned above have same time complexity n2,

there is a difference in the number of classes formed and,

hence, in the retrieval performance. The auto classification

method results in the least number of classes, and hence, the

number of leaf nodes.

Result of any of these classification schemes is an un-

ordered binary tree with following properties:

The root of the tree represents the mean of all of the

images in the database.

Each of its two children represent classified images

such that the images in the same child are maximally

correlated. Subsequent iterations are represented by

the addition of more nodes to the tree.

The internal nodes of the tree represent mean image of

the images in their children.

The leaf nodes contain maximally cross correlated

classified images whereas the number of leaf nodes

represent the number of distinct classes.

Arguably, the classification process in our scheme is the

most time consuming process since the cross correlation of

all of the images in the database against those which have

already been classified needs to be determined. However,

it is important to note that the images are classified only

once and the classification process could be done off-line.

Further, to avoid repetitive computations, at each level of

processing, we store the mean of the classes as well. Dur-

ing the retrieval phase, only these mean images are used to

make a proper selection against the given query image as

shown in Figure 1.

As mentioned earlier, the classification process recur-

sively groups images in two classes such that the images in

the same class have maximum correlation with each other

whereas the correlation is minimum between the classes.

The recursive process stops only when a threshold conditionspecified at the beginning is satisfied. It is important to note

that the threshold can be defined either in terms of number

of images in a class or a specific value of the correlation co-

efficient. The corresponding image retrieval process works

as follows:

1. Get the query image and recursively compare it with

the mean images stored in the nodes of the tree.

2. Retrieve the class having maximum correlation with

the query image.

The retrieval in this system is very efficient since thequery image is compared only with the mean images of the

classes only rather than all of the images in the database.

If the resulting binary tree is kept as an ordered tree, the

worst case performance of the retrieval process is only

O(log2(n)).


0-7695-2542-3/06 $20.00 2006IEEE


5/7

Figure 4. Sample images from the database.

4. Experimental Results

We have tested our approach on an image database con-

taining 5000 images. Our database is essentially a sub-

set of the A Very Large Image Database by the Labora-

tory for Engineering Man/Machine Systems (LEMS) at the

Brown University, Rhode Island, USA [14]. Images in this

database are binary images from a variety of different cat-

egories such as animals, people, arts, etc. An example of

images in this data collection and used in our experiments

is shown in Figure 4.

For the purpose of classification, we trained our system

and obtained statistics using all of the 5000 images in our

data collection that are size normalized. Results of classi-

fication were collected by employing the above mentionedclassification methods.

Figure 5. Mean Image of the class using auto

classification approach.

The proposed system works more efficiently if the resul-

tant binary tree is a skewed tree. In this case, retrieval will

require at most 2h iterations to retrieve relevant subset ofimages where h is the height of the tree.

Figure 2 is an example of selective classification pro-

Figure 6. A sample sub-class obtained usingauto classification approach

cess involving only 100 random images from the database

whereas the result of classification of same 100 images us-

ing auto correlation approach is shown in Figure 3. It is

important to note that with only 100 images in the two clas-

sifications schemes, the number of leaf nodes and the mean

images are roughly the same. However, it is not the case

when all of the images are classified.

Figure 5 is an example of one of the mean images repre-

senting a class in the auto classification process. The actual

images corresponding to this mean image are shown in Fig-ure 6. As can be seen, this class contains two major groups

of the images as determined by the threshold of the sys-

tem. If the value of is readjusted, the two groups will be

classified further as shown in Figure 7.

The number of leaf nodes as a function of number of

images in the database for the three mentioned classification

schemes is given in Figure 8. As one might expect, the

number of classes and the leave nodes are more in the linear

classification scheme than those in the selective and the auto

classification schemes and, hence, it may take more to both

classify images as well retrieve relevant images from the

database.

Figure 9 shows the precision-recall graph for our re-trieval results. The query image used in some of these cases

is part of the database and in some other cases, it is a totally

different image. As one can observe from the graph, the av-

erage prcission achieved for these two cases is about 90%.

It should be observed that, on the average, about 90% of the


0-7695-2542-3/06 $20.00 2006IEEE


6/7

(a)

(b)

Figure 7. Resultant sub-classes (a) and (b) af-

ter adjusting for the class in Figure 6.

relevant images can be retrieved by the system.

5. CONCLUSIONS

To estimate the degree of correlation between two given

patterns, auto correlation and cross correlation are the com-

mon statistical techniques used in the areas of image pro-

cessing and pattern recognition. In this paper, we have pre-

sented a new image classification and retrieval approach

that is based on the concept of correlation. In this ap-

proach, images are classified through an off-line process on

the basis of their cross correlation with other images in the

database. Images with maximum cross relation are recur-

sively grouped in the same class. The resultant hierarchy

is maintained as a binary tree in which each root node rep-

resents the mean of the images in its subtrees such that the

leaf nodes contain maximally correlated images, thus, mak-

ing the retrieval process very efficient.

We are well aware of the fact that the classification pro-

cess in our scheme is very expensive and has time complex-

ity of(n2

) where n is the number of images in the database.However, once classified, retrieval becomes very efficient

and requires only O(log2(n)) comparisons.At this point, we have not tested our system for insertion

of new images. Since insertion will involve recomputation

of the mean images and reclassification, it is expected to be

0

50

100

150

200

250

0 500 1000 1500 2000 2500 3000 3500 4000 4500 5000

No. of images

No.ofleafnodes

..

Auto Classification Selective Classification Linear Classification

Figure 8. Graph showing the Comparison of

Selective classification and Auto classifica-

tion.

0.00

0.10

0.20

0.30

0.40

0.50

0.60

0.70

0.80

0.90

1.00

0.00 0.20 0.40 0.60 0.80 1.00

Recall

Precision

.

Figure 9. Precision-recall graph for the aver-

age case

as expensive as the original classification process. Due to

this limitation, the proposed system may only be suitable for

those limited environments which deal with batch updates

such as the library archives.

References

[1] I. Ahmad and W. I. Grosky. Indexing and retrieval of images

by spatial constraint. Journal of Visual Communication and

Image Representation, 14(3):291320, Sept. 2003.

[2] L. G. Brown. A Survey of Image Regisration Techniques.

ACM Computing Survey, 28(4):325372, 1992.

[3] R. Brunelli and O. Mich. Image Retrieval by Example. IEEE

Transactions on Multimedia, 3(2):164171, 2000.

[4] M. Flickner, H. Sawhney, W. Niblack, J. Ashley, Q. Huang,

B. Dom, M. Gorkani, J. Hafner, D. Lee, D. Petkovic,

D. Steele, and P. Yanker. Query by Image and Video Con-

tent: The QBIC System. IEEE Computer, 28(9):2332,

Sept. 1995.


0-7695-2542-3/06 $20.00 2006IEEE


7/7

[5] T. Gevers and A. Smeulders. Pictoseek: Combining color

and shape invariant features for image retrieval. IEEE Trans-

actions on Image Processing, 9(1):102119, 2000.

[6] S. Kruger and A. Calway. Image Registration using Mul-

tiresolution Frequency Domain Correlation. In Proceedings

of the British Machine Vision Conference, pages 316325,

1998.

[7] K.-C. Lee, J. Ho, M.-H. Yang, and D. Kriegman. Vi-

sual Tracking and Recognition using Probabilistic Appear-

ance Manifolds. Computer Vision and Image Understanding

(CVIU), 99(3):303331, 2005.

[8] Z. C. M. Li, W. Liu, and H. J. Zhang. A Statistical Cor-

relation Analysis in Image Retrieval. Pattern Recognition,

35:26872693, 2002.

[9] A. Machado, C. N. J. Marinho, M. Fernando, and M. Cam-

pos. An image retrieval method based on factor analysis. In

Proc.of XVI BrazilianSymp. on Computer Graphics and Im-

age Processing (SIBGRAPI03), pages 191196, Oct. 2003.

[10] S.-H. Park and H. J. Sung. Correlation-based image regis-

tration for applications using pressure-sensitive paint. AIAA

Journal, 43(1), Jan. 2005.

[11] A. Pentland, R. W. Picard, and S. Sclaroff. Photobook: Tools

for Content-Based Manipulation of Image Databases. In

Storage and Retrieval for Image and Video Databases II,

volume 2185, pages 3447. SPIE, 1994.

[12] A. W. M. Smeulders, M. Worring, S. Santini, A. Gupta, and

R. Jain. Content-based image retrieval at the end of the early

years. IEEE Transaction on Pattern Analysis and Machine

Intelligence (PAMI) , 22(12):13491380, Dec. 2000.

[13] J. R. Smith and S.-F. Chang. VisualSEEk: A Fully Auto-

mated Content-Based Image Query System. In Proceedings

of ACM Conference on Multimedia, pages 211218, 1996.

[14] URL. http://www.lems.brown.edu/dmc, 2006.

[15] B. M. Wei-Ying Ma. NeTra: A Toolbox for Navigating

Large Image Database. Multimedia Systems, 7(3):184198,

1999.


0-7695-2542-3/06 $20.00 2006IEEE

Documents

Talal_Imran