Interactive MattingChristoph Rhemann

Supervised by:Margrit Gelautz and Carsten Rother

Matting and compositing

Matting and compositing

Outline

Talk Outline:

• Introduction & previous approaches

• Our matting model

• Evaluation strategy

= +

Cr,g,b = α Fr,g,b + (1 - α) Br,g,b

● ●

● ●

Inverse process of compositing:

Determine: F, B, αGiven:C

Matting is ill posed

= +

Underconstrained problem:7 Unknowns in only 3 Equations

● ●

Cr = α Fr + (1 - α) Br

Cg = α Fg + (1 - α) Bg

Cb = α Fb + (1 - α) Bb

Cr,g,b = α Fr,g,b + (1 - α) Br,g,b● ●

Matting is ill posed

Trimap

Scribbles

Background

Background

Unknown

Foreground

Unknown

Foreground

User interaction

Previous approaches

C = α F + (1 – α) B● ●Recall compositing equation:

Previous approaches

C = α F + (1 – α) B● ●Recall compositing equation:

Closed Form Matting [Levin et al. 06]

R

B

G

Previous approaches

C = α F + (1 – α) B● ●Recall compositing equation:

R

B

G

Closed Form Matting [Levin et al. 06]Assumption: F and B colors in a local window lie on color line

Previous approaches

C = α F + (1 – α) B● ●Recall compositing equation:

R

B

G

Closed Form Matting [Levin et al. 06]Assumption: F and B colors in a local window lie on color line Analytically eliminate F,B. Alpha can be solved in closed form

Result of [Levin et al 06]True Solution Input image + Trimap

Result of Closed Form Matting [Levin et al. 06]:• Result imperfect: Hairs cut off• Problem: Cost function has large solution space

Previous approaches

What are the reasons for pixels to be transparent?

Segmentation – based matting

Defocus Blur

Lens Camera sensor

Point spread

function

Point Spread Function

Focal plane

Lens’ aperture

Lens and defocus

Slides by Anat Levin

LensObject Camera sensor

Point spread

function

Lens’ aperture

Focal planeSlides by Anat Levin

Lens and defocus

Point Spread Function

What are the reasons for pixels to be transparent?

Segmentation – based matting

Defocus Blur Motion Blur

PSF forMotion Blur

What are the reasons for pixels to be transparent?

Segmentation – based matting

Defocus Blur Motion Blur

Discretization

What are the reasons for pixels to be transparent?

Observation: Apart from translucency mixed pixels are caused by camera’s Point Spread Function (PSF)

Segmentation – based matting

Defocus Blur Motion Blur

Discretization Translucency

Basic idea:Model alpha as convolution of a binary segmentation with PSF

Approach taken [Rhemann et al. 08]:Use this model as prior in framework of [Levin et al. 06]

Model for alpha

Binary segmentation PSF Observed alphaInput image + Trimap

Matting process

Initial alpha using [Wang et al. ´07]

(Result is imperfect)

Initialize PSF/deblur alpha

Deblured (sparse) alpha

Binarized (sparse) alpha using gradient

preserving MRF prior

Iterate a few times

Input image

Matting process

Binarized (sparse) alpha using gradient

preserving MRF prior

Segmentation prior

Final alpha

Ground truth

Result for [Levin et al. ’06]

Input image

Input image + trimap

Comparison

Result of [Wang et al. ’07]

Input image

Input image + trimap

Comparison

Input image

Input image + trimap

Result of [Rhemann et al. ’08]

Comparison

Input image + trimap [Levin et al. ’06]

[Wang et al. ’07] [Rhemann et al. ’08] Ground truth alpha

[Levin et al. ’07]

Comparison – Close up

Evaluation of matting algorithms

How to compare performance of algorithms?

Showing some qualitative results

OR

Quantitative evaluation using reference solutions

Evaluation of matting algorithms

• Key Factors for a good quantitative evaluation

• Ground truth dataset

• Online evaluation

• Perceptual error functions

• 35 natural images• High resolution• High quality

Triangulation Matting [Smith, Blinn 96]- Photograph object against 2 different backgrounds True solution to matting problem

Input image Ground truth Zoom in

Ground truth dataset

Data and evaluation scripts online

Advantages:• Investigate results• Upload novel results

www.alphamatting.com

Online evaluation

Motivation:Simple metrics not always correlated with visual quality

Input image Zoom in Result 1SAD: 1215

Result 2SAD: 806

Perceptually motivated error functions

Develop error measures for two properties:• Connectivity of foreground object• Gradient of the alpha matte

Perceptually motivated error functions

Input image Zoom in Result 1SAD: 312

Result 2SAD: 83

User Study:• Goal: Infer visual quality of image compositions• Task: Rank to according to how realistic they appear

Perceptually motivated error functions

Gradient artifacts Connectivity artifacts

Correlation of error measures to average user ranking

Gradient data Connectivity data0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8Grad.

Grad.

Conn.

Conn.

GradientConnectivitySADMSE

Correlation

Perceptually motivated error functions

• Model for alpha overcomes ambiguities

• Model-based algorithm: Performs better than competitors

• Perceptual motivated evaluation

• Message to you: Evaluation of your algorithm is important• Use ground truth data to make quantitative comparisons• Use a large dataset• Use a training / test split

Conclusions

Previous approaches

C = α F + (1 – α) B● ●Recall compositing equation:

R

B

G

Model of F

Model of B

Observed color

Data driven approaches (e.g. [Wang et al. 07])• Model color distribution of F and B (from the user defined trimap)• Observed color more likely under F or B model?• Use likelihood in framework of [Levin et al 06]

Result of data driven approaches [Wang et al. 07]:• Hair is better captured• Many artifacts in the background

Previous approaches

Result of [Levin et al 06]True Solution Input image + Trimap Result of [Wang et al 07]