1

Optimization Problem Based on L2,1-norms

Xiaohong Chen 19-10-2012

2

Outline Efficient and robust feature selection via

joint l2,1-norm minimzation Robust and discriminative distance for m

ulti-instance learning Its application…

3

Outline Efficient and robust feature selection via

joint l2,1-norm minimization Robust and discriminative distance for

multi-instance learning Its application…

4

Efficient and robust feature selection via joint l2,1-norm minimzation

5

Robust Feature Selection Based on l21-norm

Given training data {x1, x2,…, xn} and the associated class labels {y1,y2,…, yn}

Least square regression solves the following optimizaiton problem to obtain the projection matrix W

Add a regularization R(W) to the robust version of LS,

6

Robust Feature Selection Based on l21-norm

Possible regularizations

Ridge regularization

Lasso regularization

Lasso regularization

Penalize all c regression coefficients corresponding to a single feature as a whole

7

Robust Feature Selection Based on l21-norm

8

Robust Feature Selection Based on l21-norm

Denote

(14)

9

Robust Feature Selection Based on l21-norm

Then we have

(19)

10

The iterative algorithm to solve problem (14)

Theorem1: The algorithm will monotonically decrease the objective of the problem in Eq.(14) in each iteration, and converge to the globaloptimum of the problem.

11

Proof of theorem12 2

2 2

a ba

b b 2 22ab a b

2 2

2 2

a ba b

b b

u u

12

Proof of theorem1

13

(1)

(2)

(1)+(2)

14

Outline Efficient and robust feature selection via

joint l2,1-norm minimization Robust and discriminative distance for

multi-instance learning Its application…

15

Robust and discriminative distance for multi-instance learning

16

Multi-instance learning

多示例学习中，训练集由若干个具有概念标记的包 (bag) 组成，每个包包含若干个没有概念标记的示例。若一个包中至少有一个正例，则该包被标记为正 (positive) ，若一个包中所以示例都是反例，则该包被标记为反 (negative), 通过对训练包的学习，希望学习系统尽可能正确地对训练集之外的包的概念标记进行预测。

17

The illustration of MIL

18

Notations

Given N training bags and K conceptual classes.

Each bag contains a number of instances

Given the class memberships of the input data, denoted as

19

Notations

First, we represent every class as a super-bag that comprises the

instances of all its training , where

20

Objective to learn class specific distance metrics

For a given class, Ck,, we solve the following optimization problem:

21

Algorithm and its analysis

22

Algorithm and its analysis

23

Algorithm and its analysis

24

Algorithm and its analysis

On the other hand,

25

Algorithm and its analysis

26

Algorithm and its analysis

27

Algorithm and its analysis

Therefore, the objective value of the problem of (6) is decreased in each iteration till convergences.

28

Outline Efficient and robust feature selection via

joint l2,1-norm minimzation Robust and discriminative distance for m

ulti-instance learning Its application…

29

Its application

, 2,1

2,1,

( )( )

min min( )( )

i j

i j

T Ti j i j

x x

T T Wi j i j

x x

W x x x x WAW

W x x x x W BW

同类

不同类

For example:

2,1

2,1 2,1

minW

AW

BW CW

30

[1]F.Nie, D.Xu, X.Cai, and C.Ding. Efficient and robust feature selection via

joint l2,1-norm minimzation. NIPS 2010.

[2] H.Wang, F.Nie and H.Huang. Robust and discriminative distance for multi-

instance learning, CVPR 2012: 2919-2924

Reference

31

Thanks! Q&A