Silhouette Segmentation in Multiple Views

Silhouette Segmentation in Multiple Views

Wonwoo Lee, Woontack Woo, and Edmond Boyer

PAMI, VOL. 33, NO. 7, JULY 2011

Donguk [email protected]

2012.10.13

mailto:[email protected]

2Intelligent Systems

Lab.

IntroductionExtraction of consistent foreground regions from multi views without a priori knowledge of the back-groundTwo assumptions for proposed method

The region of interest appears entirely in all images.Background colors are consistent in each image.

<Approach outline>


Lab.

Probabilistic modelVariables and their dependencies

: a color image map: a binary silhouette map: the prior knowledge about the model: the foreground occupancy: the background colors: th image: a pixel located at in an image: the color value of the pixel in the th image

<The variables in different views><Dependency graph of the image (Bayesian network)>


Lab.

Joint probabilityThe joint probability of all the variables

, , and : the prior probabilities of the scene, the foreground, and the background (uniform distribution): the silhouette likelihood that determines how likely is a silhouette given the foreground shape. (spatial consistency): the image likelihood term that models the relationship between the image observation. (colors and the background informa-tion)

Independent of background colors and foreground shape

| |

| , | , ,

Pr S,F,B,I,t Pr Pr B Pr F

Pr S F Pr I B S

(1)

,

,

| , | ,

| , , | , ,

ix x

i x

i i ix x x

i x

Pr S F Pr S F

Pr I B S Pr I B S


Lab.

Spatial consistency term(1/4): the probability of a silhouette S knowing the fore-ground shape FA spatial consistency term

To evaluate the silhouette consistency between viewpoints

Using the silhouette calibration ratioThe definition of silhouette set

Using a visual hull which is the maximal volume consistent with all silhouettes

| , | ,i i j ix xPr S F Pr S S


Lab.

Spatial consistency term(2/4)

<The silhouette consistencies of pixels in image >


Lab.

Spatial consistency term(3/4)The silhouette calibration ratio at pixel

A discrete measure based on the intersections be-tween the viewing ray at and the viewing cones from other viewpoints

: a normal distribution: the highest consistency value

, : the number of views: a normalization factor: controls how influences the silhouette consistency term

Using a value of 0.7 for

2 2/1 mxC C

xR ec

(2)


Lab.

Spatial consistency term(4/4)Spatial consistency term at given pixel location

: a uniform distributionThe silhouette information at that pixel

0: background1: foreground

, 0| ,

1

ii j i b xx j i

x x

P if SPr S S

R if S

(3)


Lab.

Image likelihood termThe image likelihood term

Similarity between a pixel color and the background infor-mation (the background color model at that location)

: the statistical model of the background colorsk-component Gaussian mixture model(GMM)

: the normal distribution with mean vector and covariance matrix

: controls the threshold between foreground and back-ground assignments and ranges from 0 to 1 (uniform distribution)

, 0| , ,

1

i iB x xi i i

x x j if x

H I if SPr I B S

P if S

(4)

| ,i iB x k x k k

k

H I w N I m (5)


Lab.

Inference of the silhouettesThe probability of the silhouette at pixel

0,1

0,1

| , , ,

, , , ,

, , , ,

| , | , ,

| , | , ,

ix

ix

i j i i ix x

i j i i ix x

i j i i ix xS

i j i i i ix x x

i j i i i ix x xS

Pr S S B I

Pr S S B I

Pr S S B I

Pr S S Pr I B S

Pr S S Pr I B S

(6)


Lab.

Iterative silhouette estimationTwo assumptions

(1) Any foreground element has an appearance dif -ferent from the background in most images so that color segmentation positively detects the element in most images.

(2) The region of interest appears entirely in all of the images considered.

Iterative opitmization(1) Silhouettes are estimated using foreground and

background models. (spatial and color consis-tencies)

(2) These model are updated with the new silhou-ettes.


Lab.

InitializationForeground scene: observed by all cameras

Belongs to the 3D space region that is visible from all cameras

The background color model


Lab.

Iterative optimization via Graph cut(1/2)Iteration

(1) Estimate each silhouette using (6) with the cur-rent background models and the other current silhouettes .

(2) Update each with pixels outside the current .

For the first stepTo decide for the pixel labeling into foreground or background in each image (from equation (6))

Graph-based approaches which account for additional spatial coherence in the image


Lab.

Iterative optimization via Graph cut(2/2)Minimization of energy of the pixel assignment in

image

: the data term that measures how good pixel label is with respect to the image observation.

: the smoothness term that favors consistent labeling in homogeneous region: the set or neighboring pixel pairs in image based on 8-connectivity

: the euclidean distance

,

| , ,

| , , ,i i

i ix y

i j i i it

i j i i i i id x x s x y

x I x y NS S

E S S B I

E S S B I E I I

(7)

| , , | , , ,i j i i i i j i i id x x x xE S S B I Pr S S B I

1,

1 ,i i

s x y i ix y

E I ID I I


Lab.

Silhouette refinement

: color model for foreground: Gaussian Mixture Model (GMM)

, 0

| , ,1

i iB x xi i i

x x i j iF x x

H I if SPr I B S

H I if S

(8)

(a) Input image. (b) Silhouette after the iterative optimization(c) Silhouette after refinements


Lab.

Experimental resultsSynthetic data

Kung-fu girl data set: http://www.mpi-inf.mpg.de/departments/irg3/kungfu/

http://www.mpi-inf.mpg.de/departments/irg3/kungfu/


Lab.

Kung-fu girl data (1/2)Segmentation results with different numbers of views (top row) accounting for spatial consistencies (bottom row)

Only using background color consistency

Two views four views Six viewsOne view


Lab.

Kung-fu girl data (2/2)Proposed method


Lab.

Real data(1/7)Basic calibrationGPU-based SIFT

<Segmentation results with the Dancer data (eight views) >


Lab.

Real data(2/7)

<Segmentation results with single-object scenes>

(10 views)

(12 views)

(5 views)

(12 views)

(8 views)

(6 views)

(8 views)


Lab.

Real data(3/7)Quantitative evaluation

: the number of pixels in a set: the label set a pixel (a: the labeling F or B, b: the ground truth label)

Table 1. Silhouette extraction performance measurements

FF

F FF B

BF

F BF F

N WHit Rate

N W N W

N WFalse Alarm Rate

N W N W

(9)


Lab.

Real data(4/7)

(a) Only one object is spatially consistent. (6 views)

(b) All three objects are spatially consistent. (6 views)


Lab.

Real data(5/7)<Convergence of the extracted silhouettes: the average false alarm rates at each iteration>

<Silhouette extraction with different >


Lab.

Real data(6/7)

<Silhouette extraction for a different number of views>


Lab.

Real data(7/7)<Silhouette extraction in the presence of noise>


Lab.

ConclusionsA novel method for extracting spatially consistent silhouettes of foreground objects from several view-points

Using spatial consistency and color consistency constraints in order to identify silhouettes with unknown backgrounds

The assumptionForeground objects are seen by all images and they present color differences with the back-ground regions.


Lab.

Thank you!!!


Lab.

Silhouette calibration ratioThe silhouette calibration ratio

: an interval along ray where image contributes rays and images : the number of image contributing inside that in-terval

1 max1 i

r

ir

wr i

C N wm n


Lab.

Visual hull


Lab.

Epipolar geometry3D point

Center of projection of camera

Projection of point X

Epipole

Epipolar plane

Epipolarline

Documents

Silhouette Segmentation in Multiple Views