Fast Interactive Image Segmentation by Discriminative Clustering

Nokia Research Center

1

Fast Interactive Image Segmentation by Discriminative Clustering

Dingding Liu * Kari Pulli † Linda Shapiro * Yingen Xiong † † Nokia Research Center, Palo Alto, CA 94304, USA*Dept. Elect. Eng., University of Washington, WA 98095, USA


2

Research Aim• Cut out an object from its background fast

• Computation time – so can quickly iterate• With as few strokes as possible


3

Overview• Introduction

• Motivation • Related work

• Algorithm• Pre-segmentation by the Mean-Shift algorithm• Merge regions by discriminative clustering• Local neighborhood region classification and pruning

• Experiments and Results• Conclusions and Future Work


4

Introduction• Motivation: Image editing on mobile devices

• Convenience• Anytime, anywhere

• Challenges• Limited computational resources• Smaller screens and imprecise input


5

Related Work -Interactive Image Segmentation• Lazy Snapping

• Li et al., ACM Transactions on Graphics 2004


6

Related Work -Interactive Image Segmentation• Interactive Image Segmentation by Maximal Similarity Based Region Merging

• Ning et al., Pattern Recognition 2010

Insufficient user inputs

Sufficient user inputs


Algorithm: Summary

7

1. Pre-segmentation by the Mean-Shift algorithm2. Merge regions by discriminative clustering3. Local neighborhood region classification and

pruning


8

Background: Mean-Shift Segmentation

http://www.caip.rutgers.edu/~comanici/MSPAMI/msPamiResults.htmlhttp://robots.stanford.edu/cs223b04/CS%2520223-B%2520L11%2520Segmentation.ppt

http://www.caip.rutgers.edu/~comanici/MSPAMI/msPamiResults.html

http://www.caip.rutgers.edu/~comanici/MSPAMI/msPamiResults.html

http://robots.stanford.edu/cs223b04/CS%20223-B%20L11%20Segmentation.ppt




The Basic Mean-Shift Algorithm

9

1. Choose a search window size2. Choose the initial location of the search window3. Compute the mean (centroid of the data) within the search window4. Center the search window at that mean location5. Repeat 3 and 4 until convergence

The mean shift algorithm seeks the“mode” or point of highest density of a data distribution


Mean-Shift Segmentation

10

1. Convert the image into tokens (via color, gradients, texture measures, etc.)2. Choose initial search window locations uniformly in the data3. Compute the mean shift window location for each initial position4. Merge windows that end up on the same “peak” or mode5. Repeat 3 and 4 until convergence


11

Mean-Shift Segmentation Results


12

Algorithm: Pre-segmentation using Mean-Shift

• Three reasons for choosing the Mean-Shift algorithm:

Pre-segmentation can be done either before or after the user input

1. It preserves the boundaries better than other methods

2. Its speed has been improved significantly in recent years

3. Fewer parameters to tune


13

Algorithm: Merge non-ambiguous regions

Create two kd-trees in CIELab color space• One for the marked foreground, another for the background regionsFor each unmarked region, find the color difference to• the most similar marked background db and foreground region df

df > dthresh + db, backgrounddf + dthresh < db, foregroundOtherwise, ambiguous regionsOnly consider color, not location

Choice of dthresh :• Min difference of mean colors between the marked foreground and

background• that is higher than a minimum allowed distance (we chose 2)


Algorithm: Assign ambiguous regions

14

Now use also location informationEach of the remaining ambiguous regions is assigned • the label of the neighboring region that has the most similar mean colorIf the most similar neighboring region is also an unmarked region• merge them to a new unmarked region, repeat the processIf there is a tie in the mean color for assignment to foreground and background• the label of the region that has the most similar color variance is used


Algorithm: Prune / flip isolated regions

15

Find isolated foreground or background regions (use connected components)

Regions are changed to the opposite label when all of the following hold:

(a)The region is not marked by the user (b) The region is not the biggest region with that label (c) The region is smaller than its surrounding regions


16

Results: Segmentation time – in numbers


17

Results: Segmentation time – as a graph

0

10

20

30

40

50

60

70

80

carsten cheetah babyp cow mushroom bird goat (failure case)

time(

s)

different images

proposed algorithm

MSRM

graph cut


18

Why are we faster?• Two main reasons

• No iterative steps in the first stage• and not too many in the second or third• do the easy choices quickly

• fast nearest-neighbor lookups with kd-trees• graph-cut on many regions is slow, MSRM iterates unnecessarily much

• Merging the region descriptor is fast • only mean and standard deviation of colors• MSRM has complicated 4K bin color histograms to merge


19

Results: Segmentation time on phone


20

Results: The best segmentation quality

(a) Input image

(b) Graph-cut over regions

(c) Maximal Similar Region Merging

(d) Proposed method


21


(a) Input image



(d) Proposed method


22


(a) Input image



(d) Proposed method


23

Results: Video Demo


24

Conclusions and Future Work• A new region-based interactive image segmentation algorithm

• Significantly increases the speed of segmentation• by avoiding global optimization and long iterations

• Does not compromise the segmentation quality• Uses a region mean color instead of a single pixel color

• Future Work• Further decrease the users input• Combine the individual pixel information to further improve the

algorithm


Thank you!Questions?

Documents

Fast Interactive Image Segmentation by Discriminative Clustering