DocumentCg

Understanding and Improving the Realism of Image Composites

Aliya Ibragimova

University of Fribourg

Agenda

•  Realis7c Image composi7ng •  Iden7fying key sta7s7cs •  Human subject experiments •  Algorithm •  Results

Image composi7ng

Composi7ng procedure

Foreground Alpha maAe

New background

Composite

1.

2.

Color Transfer Technique (CTT) Reinhard et al. 2001

‘Match color’ feature of Photoshop

Color transfer technique: limita7ons

Conflates the effects of reflectance and illumina7on

Cut-‐and-‐paste Match Color

Improvements of CTT (ColorComp) Lalonde and Effros 2007

•  Analyze huge dataset of natural images : difference in distribu7on of realis7c and unrealis7c images

•  Recolor regions for realis7c composi7ng Limita7ons: requires a large dataset, depends on the presence of images that are similar to the target

Professional compositors

•  Isolate highlights and match their colors and brightness

•  Balance mid-‐tones with gamma correc7on •  Match the shadow regions

Shadows Midtones Highlights

# of pixels

Brightness

Color Harmony Cohen-‐Or et al. 2006

Harmonic colors are sets of colors that are aesthe7cally pleasing in terms of human visual percep7on.

Color Harmony Cohen-‐Or et al. 2006

Limita7ons: obtained images are not necessary realis7c, ignores luminance and contrast, the approach has not been quan7ta7vely evaluated

Alterna7ve to alpha maAe: seamlessly blending

•  Feathering •  Laplacian pyramids [Odgen et al. 1985] •  Gradient-‐domain composi7ng [Perez et al. 2003]

Alterna7ve to alpha maAe: seamlessly blending

Limita7ons: 2 source images should have similar colors and textures

Cut-‐and-‐paste Gradient-‐domain

foreground background

Problem statement

•  Which sta7s7cs control realism? •  How do these sta7s7cs affect human judgment of realism?

•  Automa7c algorithm to improve realism?

Good sta7s7cs

•  Highly correlated between foreground and background

•  Easy to adjust •  Independent from each other

Categories of sta7s7cal measures

•  Luminance •  Color temperature (CCT) •  Satura7on •  Local contrast •  Hue (circular sta7s7cs)

Sta7s7cal measures

•  Mean •  Standard devia7on •  High •  Low •  Kurtosis •  Entropy

# of pixels

Brightness mean

Standard devia7on

Find correla7on

•  Pearson correla7on coefficient •  Standard devia7on of offset δi = Mi

f – Mib

M – measure f – foreground b – background i – sta7s7cs

Sta7s7cal experiment

•  Use large (4126 images) labeled dataset •  Select the most correlated sta7s7cs

Luminance

Color Temperature

Satura7on

Sta7s7cal experiment

Results: sta7s7cal experiment •  luminance, color temperature, satura7on, local contrast are most correlated

•  mean of zones correlate more than other sta7s7cal measures

•  mean of high and low zones correlate more than mean of en7re histogram

Human subjects experiment Experiment with human subjects on Amazon Mechanical Turk (MTurk)

•  20 natural images •  3 key sta7s7cs (luminance, color temperature, satura7on)

Human subject experiments

Background luminance Foregrou

nd luminance

Human subjects experiment

Human subjects experiment

Automa7c composite adjustment

•  Zone selec7on using machine learning (random forest classifier) T = s * σg, s = 0.1 Three binary classifiers:

•  Pick smallest changes if mul7ple zones •  Features for foreground and background and per sta7s7c: std, skew, kurt, entropy, p1, p2 …p20.

Pipeline Input: foreground and background image 1.  Match H-‐zone of contrast using S-‐shape 2.  Select zone and adjust mean of luminance 3.  Select zone and adjust mean of CCT 4.  Select zone and adjust satura7on Adjust algorithm is greedy, could iterate several 7mes if needed

Results and evalua7on

Cut-‐and-‐paste Manual MatchColor ColorComp Ours


Cut-‐and-‐paste Manual MatchColor ColorComp Ours


Results and evalua7on Thurstone’s Law of Compara7ve Judgement


•  BeAer than previous methods •  Close to manual edi7ng

Limita7ons: •  Religh7ng •  Theory for zone selec7on

Ques7ons?

Technology

DocumentCg