View
217
Download
0
Category
Preview:
Citation preview
Poisson Ma)ng Jian Sun, Jiaya Jia, Chi-‐Keung Tang, heung-‐Yeung Shum
Presenter: Vera Dadok Discussant: Eileen Bai
Recall: Ma)ng • Ma)ng equaEon over image
– F – foreground – B – background – α – alpha maKe
ResulEng alpha maKe Image I Source: hKp://www.cse.cuhk.edu.hk/~leojia/all_project_webpages/Poisson%20ma)ng/poisson_ma)ng.html
BFI )1( αα −+=
Recall: Ma)ng • Ma)ng equaEon over image
• Under-‐constrained equaEon: – 7 unknown variables
• α • R,G,B channels of F • R,G,B channels of B
– 3 known variables • R,G,B channels of I
• AddiEonal informaEon – User-‐defined Trimap
Trimap
Image I
BFI )1( αα −+=
Poisson Ma)ng Another approach to solving the ma)ng problem Approach:
– Use gradient of image to calculate gradient of maKe – Reconstruct maKe from gradient of maKe
Key addi.on: – Enables user interacEon to locally improve results
Poisson Ma)ng • Ma)ng equaEon
• Gradient of ma)ng equaEon
• Gradient rewriKen
• Once gradient of α is obtained, we can create maKe
BFBFI ∇−+∇+∇−=∇ )1()( ααα
BFI )1( αα −+=
A = 1F −B
D =α∇F − (1−α)∇B))1((1 BFI
BF∇−−∇−∇
−=∇ ααα
Poisson Ma)ng • Ma)ng equaEon
• Gradient of ma)ng equaEon
• Gradient rewriKen
• Once gradient of α is obtained, we can create maKe
)( DIA −∇=∇α
BFBFI ∇−+∇+∇−=∇ )1()( ααα
BFI )1( αα −+=
A = 1F −B
D =α∇F − (1−α)∇B
Poisson Ma)ng: Two steps
• Step 1: Global (required) – Solve global Poisson ma)ng problem, assumes
• Step 2: Local (opEonal) – Adjust local regions with local Poisson ma)ng operaEons
(1) Global Poisson Ma)ng (2) Local Poisson Ma)ng
)( DIA −∇=∇α
D = 0
Step 1: Global Poisson Ma)ng
Global soluEon assumes D = 0 in gradient equaEon
– i.e. assumes “smooth” foreground and background
IntuiEvely – MaKe gradient is approximately proporEonal to image gradient
)( DIA −∇=∇α
A = 1F −B
D =α∇F − (1−α)∇B
∇α ≈∇IF −B
D =α∇F − (1−α)∇B = 0
General Case Global Case with AssumpEon
What does ‘smooth’ mean? D =α∇F − (1−α)∇B = 0
BFBFI ∇−+∇+∇−=∇ )1()( ααα
Color Image Grayscale I
Solve for alpha using smooth assump.on, D=0 What should the ma>e edge be?
What does ‘smooth’ mean? D =α∇F − (1−α)∇B = 0
BFBFI ∇−+∇+∇−=∇ )1()( ααα
Grayscale I
Gradient α (in x) Gradient I (in x)
α
∇α ≈∇IF −B
What does ‘smooth’ mean? D =α∇F − (1−α)∇B = 0
BFBFI ∇−+∇+∇−=∇ )1()( ααα
F
α
B Does this seem like a good ma>e? Was this “smooth”?
What does ‘smooth’ mean?
Color Image Grayscale I
D =α∇F − (1−α)∇B = 0BFBFI ∇−+∇+∇−=∇ )1()( ααα
What should the ma>e edge be?
What does ‘smooth’ mean?
Color Image
Does this seem like a good ma>e? Was this “smooth” assump.on good?
F
α
B
D =α∇F − (1−α)∇B = 0BFBFI ∇−+∇+∇−=∇ )1()( ααα
Step 1: Global SoluEon method This simplified equaEon is solved in grayscale channel
– 2 known variables: • gradients of I
– 3 unknown variables: • (F-‐B), treated as a single variable • gradients of α
∇α ≈∇IF −B
+ AddiEonal knowledge • Boundary condiEons from Trimap
Step 1: Global SoluEon method Minimize error in
In the sense of least squares
This has an associated Poisson equaEon with Dirichlet boundary condiEons (due to Trimap), which we know how to solve
∇α ≈∇IF −B
This is solved iteraEvely, approximaEng F-‐B, then solving for α, then repeat
Global Poisson Ma)ng
Results good for smooth F and B (thus D≈0)
Source: hKp://www.cse.cuhk.edu.hk/~leojia/all_project_webpages/Poisson%20ma)ng/poisson_ma)ng.html
Why do we need local operaEons?
Global Poisson maFng assumed D=0
Exact gradient equaEon
)( DIA −∇=∇αA = 1
F −BD =α∇F − (1−α)∇B
Step 2: Local OperaEons
Do not assume D=0 • Goal: change A, D to produce beKer approximaEon of α
– ‘Fix’ problematic areas in global result • Two categories
– Channel SelecEon – Local filters (brushes)
Δα = div(A(∇I −D))
)( DIA −∇=∇α
• Aner local α-‐gradient changes solve local Poisson equaEon with Dirichlet condiEons – Key: local changes in α-‐gradient do not produce
arEfacts in final maKe
Local OperaEons: Channel SelecEon • Create new weighted channel of R, G, B to minimize |D|
– Instead of grayscale channel – Reduces error in D
• Process – User selects background samples – AutomaEcally calculates channel weights to minimize variance of foreground and background colors
)( DIA −∇=∇α
grayscale
new weighted channel
α maKe channel
Local OperaEons: High-‐pass filter
• Low pass filter on D • High-‐pass on α due to term – D is esEmated using low-‐frequencies of gradient
before
aner
)( DIA −∇=∇α
(∇I −D)
MaKe gradient MaKe
Local OperaEons: BoosEng brush
• Increase or decrease sharpness
• Gaussian filter on A
before
aner
)( DIA −∇=∇α
MaKe gradient MaKe
Local OperaEons: Diffusion filtering
• Remove JPEG arEfacts
before
aner
)( DIA −∇=∇α
MaKe gradient MaKe
Local OperaEons: Clone Brush
• Exactly copy gradient )( DIA −∇=∇α
MaKe gradient MaKe
Erase brush Inverse brush
Full algorithm summary 0: Create Trimap (user defined) 1: Global Poisson ma)ng (automaEc) 2: Apply local Poisson operators (user defined)
(i) Apply channel selecEon (smoother background or foreground, reduces |D|) (ii) Apply highpass filtering to obtain approximaEon of D (iii) Apply boosEng brush to change A (iv) Apply clone brush to disEnguish gradients
Note: the order of these steps is important! Erase brush and inverse brush can be applied any Eme
Video break
Global SoluEon: 1:10 Local SoluEons: 1:30
Full algorithm summary
Source: hKp://www.cse.cuhk.edu.hk/~leojia/all_project_webpages/Poisson%20ma)ng/poisson_ma)ng.html
User (Trimap)
Comparison to other methods
• Via picture
Comparison to other methods
• Via picture
Improvements over other methods
• Natural image ma)ng including Bayesian Ma)ng – Not easy to adjust output of algorithms if mistakes are made in alpha maKe
• Difference ma)ng and triangulaEon ma)ng – Need mulEple input images
ApplicaEons
• Image composiEng
Other applicaEons
• De-‐fogging
Other ApplicaEons
• MulE-‐background
Thank you!
QuesEons?
Discussion
LimitaEons 1.) User-‐defined local operaEons
l Relies on the user for input l Good enough? l Background knowledge of brushes
2.) Specific order of operaEons l Restricts freedom/flexibility
3.) Similar foreground/background elements
l Hard to disEnguish underlying structure of maKe l Difficult for computers as well as humans
*First two points could be argued as posiEve qualiEes as well.
4.) MulEple disEnct foreground elements l Requires complicated trimap l Bayesian could solve relaEvely simply
Improvements
l Ideas? l Separate “hair” from “fur” - These two come up the most onen - Have very different properEes
Future Work
l Extends naturally to... Video ma)ng! - “Real-‐Time Video Ma)ng Using MulEchannel Poisson EquaEons” -‐ 2010
- AutomaEcally generated trimaps
l CombinaEon of Poisson and Bayesian Ma)ng - Combine user-‐defined local operaEons with the pixel by pixel
esEmaEon in Bayes'
Recommended