Embed Size (px)
Citation preview
IMAGE COMPLETION Mark Chang
Scene Completion Using Millions of Photographs • James Hays & Alexei A. Efros • Carnegie Mellon University • ACM Transactions on Graphics (SIGGRAPH 2007) • http://graphics.cs.cmu.edu/projects/scene-completion/
Scene Completion Using Millions of Photographs
照⽚片 輸⼊入 輸出
類似圖⽚片 圖庫 影像融合 影像檢索
使⽤用圖庫
⽤用圖⽚片中的其他地⽅方填補 ⽤用不同圖⽚片填補
原圖 輸⼊入影像
影像檢索
找到符合語意的圖⽚片 找到不符合語意的圖⽚片
影像融合
圖⽚片1
圖⽚片2 融合成功
融合失敗
詳細過程介紹 • 圖庫取得：Flickr Image • 影像檢索：Gist Descriptor • 影像融合：Graph Cut & Poisson Blending
圖庫取得 • Photo from Flickr.com • 2.3 million unique images • 396 gigabytes of JPEG compressed data
影像檢索 • 取出影像特徵（Gist Descriptor） • 找尋特徵較接近的影像（Euclidean Distance）
Euclidean Distance
Gist Descriptor
情景（Scene） • 情景相近：圖⽚片內容物語意相近 • 不需要特別辨識個別物體。
⼤大致情境 詳細內容（不符語意）
Modeling the Scene
Fourier Transform
X(!1,!2) =1X
n1=�1
1X
n2=�1x(n1, n2)e
�j(!1n1+!2n2)
Fourier Transform
Receptive Fields
Gist Descriptor
Gist Descriptor
影像遮罩
影像檢索 • 以量取勝 • 照⽚片越多，越能找到適⽤用的照⽚片
影像融合
融合⾊色調和材質 Poisson Blending
得出適當的邊界 Graph Cut
Graph Cut
合併後的圖⽚片 圖⽚片接合處 圖⽚片1 圖⽚片2
Vivek Kwatra, Arno Schodl, Irfan Essa, Greg Turk and Aaron Bobick. Graphcut Textures: Image and Video Synthesis Using Graph Cuts.
Graph Cut
cost(s, t, A,B) = |A(s)�B(s)|+ |A(t)�B(t)|
s
A(s) A(t)
B(t)B(s)
t
Graph Cut cost(s, t, A,B) = |A(s)�B(s)|+ |A(t)�B(t)|
A(s)
A(t)
B(t)
B(s)
A(s)
A(t)
B(t)
B(s)
cut
overlap
Graph Cut
cost =X
p
Cd(p, L(p)) +X
p,q
Ci(p, q, L(p), L(q))
L(p) =
(patch
exist
hole
exist patch
(Cd(p, patch) = (k ⇥Dist(p, hole))
3
Cd(p, exist) = 1 if p in hole
Graph Cut
cost(s, t, A,B) = |A(s)�B(s)|+ |A(t)�B(t)|原本的 graph-cut：融合失敗
轉換到 gradient domain：融合成功
圖⽚片1 圖⽚片2
cost(s, t, A,B) = |rA(s, t)�rB(s, t)|
Poisson Blending Patrick Perez, Michel Gangnet and Andrew Blake. Poisson Image Editing.
Poisson Blending
• v : Gradient of a region in an image • g : Selected region of source • f* : Known pixel values in domain S • f : Unknown pixel values in domain Ω • Ω : Region g placed on domain S • ∂Ω : Boundaries
min
ZZ
⌦|rf � v|2 with f |@⌦ = f⇤|@⌦
Poisson Blending : 1D Example
min((f2 � f1 � 1)2 + (f3 � f2 � (�1))2 + (f4 � f3 � 2)2
+ (f5 � f4 � (�1))2 + (f6 � f5 � (�1))2)
min(2f22 � 2f2f3 � 16f2 + 2f2
3 + 6f3 � 2f3f4 + 2f24 � 2f4f5 � 6f4 + 2f2
5 � 2f5 + 59)
with f1 = 6, f6 = 1
min
ZZ
⌦|rf � v|2 with f |@⌦ = f⇤|@⌦
Poisson Blending : 1D Example dQ
df2= 4f2 � 2f3 � 16
dQ
df3= �2f2 + 4f3 � 2f4 + 6
dQ
df3= �2f3 + 4f4 � 2f5 � 6
dQ
df3= �2f4 + 4f5 � 2
8>>><
>>>:
f2 = 6
f3 = 4
f4 = 5
f5 = 3
2
664
4 �2 0 0�2 �4 �2 00 �2 4 �20 0 �2 4
3
775
2
664
f2f3f4f5
3
775 =
2
664
16�662
3
775
source target
實驗結果 • 成功案例 • 失敗案例
成功案例⼀一
成功案例⼆二
成功案例三
失敗案例⼀一：邊界太明顯
失敗案例⼆二：找錯影像
失敗案例三：邊界錯誤
⽤用原圖材質填補，效果⽐比較好
原圖 ⽤用原圖材質填補 ⽤用其他圖⽚片填補
近期相關研究 • Deep Learning
• Supervised Semantics-Preserving Deep Hashing for Large-Scale Image Search
http://arxiv.org/pdf/1507.00101.pdf
延伸應⽤用 • 全景圖產⽣生器 • 透視圖產⽣生器
全景圖產⽣生器
Automatic Generation of An Infinite Panorama
http://cybertron.cg.tu-berlin.de/pdci09/infinite_panorama/ http://www.cs.cmu.edu/afs/andrew/scs/cs/15-463/f07/proj_final/www/lisachan/
透視圖產⽣生器
http://parosky.net/projects/undresser.html http://parosky.net/assets/data/anoron2013w.pdf
画像補完技術による衣服の除去
延伸閱讀 • Scene Completion
• James Hays & Alexei A. Efros. Scene Completion Using Millions of Photographs.
• GIST Descriptor • Aude Oliva & Antonio Torralba. Modeling the Shape of the Scene: A
Holistic Representation of the Spatial Envelope. • Aude Oliva & Antonio Torralba. Building the gist of a scene: the role of
global image features in recognition. • Graph Cut
• Vivek Kwatra, Arno Schodl, Irfan Essa, Greg Turk and Aaron Bobick. Graphcut Textures: Image and Video Synthesis Using Graph Cuts.
• Poisson Blending • Patrick Perez, Michel Gangnet and Andrew Blake. Poisson Image
Editing. • http://eric-yuan.me/poisson-blending
圖⽚片來源 • James Hays & Alexei A. Efros. Scene Completion Using Millions of Photographs.
• Aude Oliva & Antonio Torralba. Modeling the Shape of the Scene: A Holistic Representation of the Spatial Envelope.
• Aude Oliva & Antonio Torralba. Building the gist of a scene: the role of global image features in recognition.
圖⽚片來源 • Vivek Kwatra, Arno Schodl, Irfan Essa, Greg Turk and Aaron Bobick. Graphcut
Textures: Image and Video Synthesis Using Graph Cuts.
• Patrick Perez, Michel Gangnet and Andrew Blake. Poisson Image Editing.
• http://eric-yuan.me/poisson-blending
• http://d.hatena.ne.jp/LM-7/20090629/1246282979
