1 Graph Embedding (GE) & Marginal Fisher Analysis (MFA) 吳沛勳...

1

Graph Embedding (GE) &Marginal Fisher Analysis (MFA)

吳沛勳 seanpsw@gmail.com

劉冠成 zeter168@gmail.com

韓仁智 jchan@visionatics.com.tw

2

Outline

1. Introduction

2. System Flowchart

3. Dimensionality Reduction- Graph Embedding 3.1 Cost Function

-Intrinsic Graph/Penalty Graph

3.2 Linearization

3.3 Example: LDA

4. Marginal Fisher Analysis

5. Experiments Result

3

Outline

1. Introduction

2. System Flowchart

3. Dimensionality Reduction- Graph Embedding 3.1 Cost Function

-Intrinsic Graph/Penalty Graph

3.2 Linearization

3.3 Example: LDA

4. Marginal Fisher Analysis

5. Experiments Result

4

1. Introduction

We present a general framework called Graph

Embedding (GE).

In graph embedding, the underlying merits and

shortcomings of different dimensionality reduction

schemes, existing or new, are revealed by

differences in the design of their intrinsic and penalty

graphs and their types of embedding.

A novel dimensionality reduction algorithm, Marginal

Fisher Analysis (MFA).

5

Outline

1. Introduction

2. System Flowchart

3. Dimensionality Reduction- Graph Embedding 3.1 Cost Function

-Intrinsic Graph/Penalty Graph

3.2 Linearization

3.3 Example: LDA

4. Marginal Fisher Analysis

5. Experiments Result

6

2. Face Recognition Flowchart

N : # Image (200, 20 image pre person)

: # People (10)

m : Image Size (24x24)

cN

mm

200)2424(

1][ RRxxX Nm

N

37)2424( RRw mm

w : Unitary Linear Projection Vector

k : k nearest neighbors

X

1. Training Image Set: X

2.1 MFA Space Creation: w

2.2 Projection to MFA Space: Y

20037' , RRYXwY NmT

3. k-NN Classification

1. Test image: xtest

Classification Result

Training Process

Test Process

2. Projection to MFA Space: ytest

21, kk

1)2424( Rxtest

137, Ryxwytesttest

T

test

k

winfluence

w

Y

k1 : k-NN for intrinsic graphk2 : k-NN for penalty graph

7

Outline

1. Introduction

2. System Flowchart

3. Dimensionality Reduction- Graph Embedding 3.1 Cost Function

-Intrinsic Graph/Penalty Graph

3.2 Linearization

3.3 Example: LDA

4. Marginal Fisher Analysis

5. Experiments Result

8

3. Graph Embedding

For a dimensionality reduction problem, we require an intrinsic graph G and, optionally, a penalty graph as input.

We now introduce the dimensionality reduction problem from the new point of view of graph embedding.

Let G={X,W} be an undirected weighted graph (two-way direction) with vertex set X (N nodes) and similarity (or weighted) matrix .

W :

1. Symmetric matrix

2. May be negative

44

4321

RW

xxxxX

NxxxX 21

NNRW

9

3. Graph Embedding: Laplacian Matrix

L=Degree-Adjacent=>

jiijii iWD

WDL

.,(2)

0011

0001

1001

1110

W

2000

0100

0020

0003

D

2011

0101

1021

1113

L

W is weighted matrix ,also call similarity matrix

G={X,W} :

10

3.1 Cost Function (1/2) Our graph-preserving criterion is given as follows:

For larger (positive) similarity samples and :

For smaller (negative) similarity samples and :

B typically is the Laplacian matrix of a penalty graph .

Use Lagrange multipliers to solve:

LyyWyyy T

dByyijji

dByy TT minarg minarg*

2(3)

2

ji yy

2

ji yy

ixjx

ix jx

pG

ppp WDLB

YLYBy

u

dBYYLYYu TT

1

rmPenalty te

graphpenalty

tion termRegulariza

graph intrinsic

0

)(

Y must be an eigenvector of the LB 1

Intrinsic Graph

Penalty Graphunknown Known

11

3.1 Cost Function: Intrinsic Graph in LDA

2

100000

02

10000

004

3000

0004

300

00004

30

000004

3

66RD

ji

jicc

c

cc

ij

cc

cc

nW

ji

ji

0

1,

,

21

21

654321

]65[ ]4321[

2 4 2 6

cc

nnNN

xxxxxxX

c

If we have 6 images for 2

people.

2

1

2

10000

2

1

2

10000

004

3

4

1

4

1

4

1

004

1

4

3

4

1

4

1

004

1

4

1

4

3

4

1

004

1

4

1

4

1

4

3

66RL

02

10000

2

100000

0004

1

4

1

4

1

004

10

4

1

4

1

004

1

4

10

4

1

004

1

4

1

4

10

66RW

WDL is not important, because in GE i always not equal to j.iiW

12

3.1 Cost Function: Penalty Graph

We define an intrinsic graph to be the graph G itself

Penalty graph :

1. As a graph whose vertices X are the same as those

of G.

2. Whose edge weight matrix corresponds to the similarity

characteristic that is to be suppressed in the dimension-

reduced feature space .

3. Penalty graph = constraint

},{ pp WXG

pW

13

3.1 Cost Function: Penalty Graph in LDA

100000

010000

001000

000100

000010

000001

66RD p

21

21

654321

]65[ ]4321[

2 4 2 6

cc

nnNN

xxxxxxX

c

If we have 6 images for 2

people.

6

5

6

1

6

1

6

1

6

1

6

16

1

6

5

6

1

6

1

6

1

6

16

1

6

1

6

5

6

1

6

1

6

16

1

6

1

6

1

6

5

6

1

6

16

1

6

1

6

1

6

1

6

5

6

16

1

6

1

6

1

6

1

6

1

6

5

B 66R

6

1

6

1

6

1

6

1

6

1

6

16

1

6

1

6

1

6

1

6

1

6

16

1

6

1

6

1

6

1

6

1

6

16

1

6

1

6

1

6

1

6

1

6

16

1

6

1

6

1

6

1

6

1

6

16

1

6

1

6

1

6

1

6

1

6

1

66RW p

NW p

ij1

ppp WIWB DMaximize class covariance is equal to maximize data covariance.

Here, LDA penalty graph is as PCA intrinsic graph: Consider only btw-class

scatter.

14

3.2 Linearization

Assuming that the low-dimensional vector representations of the

vertices can be obtained from a linear projection as ,

where w is the unitary projection vector, the objective function in

(3) becomes :

wXLXwWxwxww TT

dwwdwXBXwji

ijjT

iT

dwwdwXBXw

T

TT

T

TT

or

2

or

* minargminarg (4)

Solution

:wLwBBwLw

w

u

dwXBXwwXLXwu TTTT

10

)(

w can be solved by singular value decomposition (SVD)

15

Outline

1. Introduction

2. System Flowchart

3. Dimensionality Reduction- Graph Embedding 3.1 Cost Function

-Intrinsic Graph/Penalty Graph

3.2 Linearization

3.3 Example: LDA

4. Marginal Fisher Analysis (MFA)

5. Experiments Result

16

4.1 Marginal Fisher Analysis (MFA) (1/3)

Fig. 4.

The adjacency relationships of the intrinsic and penalty graphs for the Marginal Fisher Analysis algorithm.

k-NN : k1=5

In same class

k-NN : k2=2

In difference class

Marginal k-NN

(Within-Class) (Btw-Class)

17

(15)wXWDXw

wXWDXw

S

Sw

TppT

TT

wp

c

w )(

)(minarg~

~minarg*

Cost function:

Minimize within-class (intrinsic graph)Maximize between- class (penalty graph)

(13)

else ,0

)(or )( if,1

2

~

11

11

2

iNjjNiW

wW)XX(Dw

WxwxwS

kkij

TT

iji (i)N(j) or jNi

jT

iT

c

kk

Intrinsic graph

4.1 Marginal Fisher Analysis (MFA) (2/3)

indicates the index set of the k1 nearest neighbors of the sample xi in the same

class.

)(1

iNk

)(1

iNk

L

k-NN : k1=5

In same class

18

4.1 Marginal Fisher Analysis (MFA) (3/3)

(14)

else ,0

)(),(or )(),( if,1

2

~

22

22

2

jkikpij

TppT

iji (cj)P) or (i,j)(cP(i,j)

jT

iT

p

cPjicPjiW

w)XWX(Dw

WxwxwSkik

Penalty graph

How to decide k1 and k2-nearest neighbor : k-nearest neighbor (k-NN)

is a set of data pairs that are the k2 nearest pairs among the set

denote the index set belonging to the cth class

)(2

cPk}j,ij),{(i, c c

c

B

19

4.1 Q&A (1/2)

Q1: MFA: How to Decide k1 k2

A1: k1 : (Within Class) Sampled five values between two and {minc(nc-1)},

and chose the value with the best MFA performance.

nc: # of images per class (subject)

(We direct use 5)

k2 : (Btw-Class) Choose the best k2 between 20 and 8Nc at sampled

intervals of 20.

Nc: # of classes (subjects)

(We direct use 20)

20

MFA: Comparison with LDA Advantages :

1. The number of available projection directions (axes) is much larger than that of LDA (MFA finds more significant axes and has better classification results. ).

MFA: Rank (B-1L) LDA: Nc-1

2. There is no assumption on the data distribution, thus it is more general for discriminant analysis, LDA assumption data is approximately Gaussian distributed.

Data distribution: MFA: Non-linear LDA: Linear

3. The inter-class margin can better characterize the separability of different classes than the inter-class scatter in LDA.

MFA: Maximize margin LDA: Difference between means

Disadvantage:

LDA-> incremental LDA, MFA->?

4.1 Q&A (2/2)

margin

Positive Negative

2121

Outline

1. Introduction

2. System Flowchart

3. Dimensionality Reduction- Graph Embedding 3.1 Cost Function

-Intrinsic Graph/Penalty Graph

3.2 Linearization

3.3 Example: LDA

4. Marginal Fisher Analysis

5. Experiments Result

22

5. Experiments: Database (1/2)

Dadabase Yale B

People 10

#Image30 per person, random select 20 image for training, and

remain 10 image for test (G30/P20)

Image size 24x24

Variations Variable illumination, cropped face

24x24

23

5. Experiments Result

k-NN k=1 k=3 k=5

ErrorRate 11.72%±3.58 11.04%±3.66 11.10±3.27

For each k run 100 times, and calculate mean and standard

deviation .

24

Reference

1. P. Belhumeur, J. Hespanha, and D. Kriegman, “Eigenfaces vs. Fisherfac

es: Recognition Using Class Specific Linear Projection,” IEEE Trans. on

Pattern Analysis and Machine Intelligence, Vol. 19, No. 7, pp. 711–720, 1

997.

2. T.K. Kim, S.F. Wong, B. Stenger, J. Kittler and R. Cipolla, “Incremental

Linear Discriminant Analysis Using Sufficient Spanning Set

Approximations”, CVPR, pp. 1-8, 2007.

3. S. Yan, D. Xu, B. Zhang, H. Zhang, Q. Yang, and S. Lin, “Graph

Embedding and Extensions: A General Framework for Dimensionality

Reduction,” IEEE Trans. on Pattern Analysis and Machine Intelligence,

Vol. 29, No. 1, pp. 40–51, 2007.