Embed Size (px)
Neural Doodle By Mark Chang
This Work
• Title: – Seman;c Style Transfer and Turning Two-‐Bit Doodles into Fine Artwork
• Author: – Alex J. Champandard
• Ins;tu;on: – nucl.ai Research Laboratory
• URL: – hKp://arxiv.org/abs/1603.01768
Previous Works
• A Neural Algorithm of Ar;s;c Style – Leon A. Gatys, Alexander S. Ecker and MaKhias Bethge – hKp://arxiv.org/abs/1508.06576
• Combining Markov Random Fields and Convolu;onal Neural Networks for Image Synthesis. – Chuan Li and Michael Wand – hKp://arxiv.org/abs/1601.04589
A Neural Algorithm of Ar;s;c Style
content
style
artwork
hKp://www.slideshare.net/ckmarkohchang/neural-‐art-‐english-‐version
Combining Markov Random Fields and Convolu;onal Neural Networks … content style artwork
Patch Based v.s Gram-‐based
gram-‐based patch-‐based content
style
Patch-‐Based Style Transfer
Width*Height
Depth style
VGG19 canvas
Width*Height Depth
Filter Responses VGG19
Most Similar Patch
Minimize the Difference
Patch-‐Based Style Transfer
Style: Width*Height
Depth �(x)
x
xs �(xs)
i(�(xs))
Filter Responses:
Patch of Filter Response:
Canvas: Width*Height
Depth i(�(x))
Filter Responses:
Patch of Filter Response:
Patch-‐Based Style Transfer
Es(�(x),�(xs)) =mX
i=1
k i(�(x))� NN(i)(�(xs))k2
NN(i) = argmini
i(�(x)) · j(�(xs))
| i(�(x))| · | j(�(xs))|
Width*Height
Width*Height
Depth
Most Similar Patch:
Style Loss Func;on:
Patch-‐Based Style Transfer
Content Loss Func;on:
Regularizer:
Ec(�(x),�(xc)) = k�(x)� �(xc)k2
Total Loss:
R(x) =X
i,j
(xi,j+1 � xi,j)2 + (xi+1,j � xi,j)
2
E = ↵Es(�(x),�(xs)) + �Ec(�(x),�(xc)) + �R(x)
Content: xc
Seman;c Style Transfer content
style
seman;c maps
artwork
Patch-‐Based With Seman;c Maps
content
style
patch-‐based only
patch-‐based with seman;c
maps
seman;c maps
Seman;c Style Transfer
content
style
seman;c maps
Pixel Labeling
Seman;c Style Transfer
Width*Height
Depth
Filter Responses: VGG19
style
seman;c map of style
Average Pooling
Width*Height
Depth
Pooling Result:
�(xs)
ms
concatenate
Width*Height
Depth
Ss = �(xs)kkms
weight of seman;cs
Seman;c Style Transfer
style
seman;c map of style
canvas
seman;c map of content
Most Similar Patch:
Style Loss Func;on:
SSs
NN(i) = argmini
i(Ss) · j(S)
| i(Ss)| · | j(S)|
Es(S,Ss) =mX
i=1
k i(S)� NN(i)(Ss)k2
Seman;c Style Transfer
Total Loss:
style
seman;c map of style
canvas
seman;c map of content
SSs
content
E = ↵Ec(�(x),�(xc)) + �Es(S,Ss) + �R(x)
xc
x
Image Analogy
Image Analogy
Total Loss:
Ss S
style seman;c
map of style canvas seman;c
map of content
No Content Loss
E = ↵Ec(�(x),�(xc)) + �Es(S,Ss) + �R(x)
Image URL
hKp://arxiv.org/abs/1601.04589
hKp://arxiv.org/abs/1603.01768
Source Code
• Neural Doodle – hKps://github.com/alexjc/neural-‐doodle
• Patch-‐Based Style Transfer – hKps://github.com/chuanli11/CNNMRF
About the Speaker
• Email: ckmarkoh at gmail dot com • Blog: hKp://cpmarkchang.logdown.com • Github: hKps://github.com/ckmarkoh
Mark Chang
• Facebook: hKps://www.facebook.com/ckmarkoh.chang • Slideshare: hKp://www.slideshare.net/ckmarkohchang • Linkedin: hKps://www.linkedin.com/pub/mark-‐chang/85/25b/847
