Deep Learning for Computer Visionvllab.ee.ntu.edu.tw/uploads/1/1/1/6/111696467/dlcv_w9__1...Chen et...

Deep Learning for Computer VisionSpring 2018

http://vllab.ee.ntu.edu.tw/dlcv.html (primary)

https://ceiba.ntu.edu.tw/1062DLCV (grade, etc.)

FB: DLCV Spring 2018

Yu-Chiang Frank Wang 王鈺強, Associate Professor

Dept. Electrical Engineering, National Taiwan University

2018/05/09

What’s to Be Covered Today…

• Visualization of NNs• t-SNE• Visualization of Feature Maps• Neural Style Transfer

• Understanding NNs• Image Translation• Feature Disentanglement

• Recurrent Neural Networks• LSTM & GRU

Many slides from Fei-Fei Li, Yaser Sheikh, Simon Lucey, Kaiming He, J.-B. Huang, Yuying Yeh, and Hsuan-I Ho

Representation Disentanglement

• Goal• Interpretable deep feature representation• Disentangle attribute of interest c from the derived latent representation z

• Unsupervised: InfoGAN• Supervised: AC-GAN

InfoGANChen et al.

NIPS ’16

ACGANOdena et al.

ICML ’17

Chen et al., InfoGAN: Interpretable representation learning by information maximizing generative adversarial nets., NIPS 2016.Odena et al., Conditional image synthesis with auxiliary classifier GANs. ICML’17 3

AC-GAN

Odena et al., Conditional image synthesis with auxiliary classifier GANs. ICML’17

Real dataw.r.t. its domain label

• Supervised Disentanglement

G∗ = arg minG

ℒ𝐺𝐺𝐺𝐺𝐺𝐺 G, D + ℒ𝑐𝑐𝑐𝑐𝑐𝑐 G, D

ℒ𝐺𝐺𝐺𝐺𝐺𝐺 G, D = 𝔼𝔼 log 1 − D(G(𝑧𝑧, 𝑐𝑐)) + 𝔼𝔼 log D 𝑦𝑦

ℒ𝑐𝑐𝑐𝑐𝑐𝑐 G, D = 𝔼𝔼 − log Dcls(𝑐𝑐′|𝑦𝑦) + 𝔼𝔼 − log Dcls(𝑐𝑐|G(𝑥𝑥, 𝑐𝑐))

• Learning• Overall objective function

• Adversarial Loss

• Disentanglement lossG

𝑧𝑧𝑐𝑐

G(𝑧𝑧, 𝑐𝑐)𝑦𝑦 (real)

Supervised

Generated dataw.r.t. assigned label

AC-GAN

Odena et al., Conditional image synthesis with auxiliary classifier GANs. ICML’17

• Supervised Disentanglement

𝑧𝑧𝑐𝑐

G(𝑧𝑧, 𝑐𝑐)𝑦𝑦 (real)

Supervised

Different 𝑐𝑐 values

InfoGAN

• Unsupervised Disentanglement

G∗ = arg minG

ℒ𝐺𝐺𝐺𝐺𝐺𝐺 G, D = 𝔼𝔼 log 1 − D(G(𝑧𝑧, 𝑐𝑐)) + 𝔼𝔼 log D 𝑦𝑦

ℒ𝑐𝑐𝑐𝑐𝑐𝑐 G, D = 𝔼𝔼 − log Dcls(𝑐𝑐|G(𝑥𝑥, 𝑐𝑐))

• Learning• Overall objective function

• Disentanglement loss

Chen et al., InfoGAN: Interpretable representation learning by information maximizing generative adversarial nets., NIPS 2016.6

InfoGAN

• Unsupervised Disentanglement

Chen et al., InfoGAN: Interpretable representation learning by information maximizing generative adversarial nets., NIPS 2016.

• No guarantee in disentangling particular semantics

Different 𝑐𝑐

Rotation Angle Width

Training process

Different 𝑐𝑐

• Representation Disentanglement• InfoGAN: Unsupervised representation disentanglement• AC-GAN: Supervised representation disentanglement

• Image Translation• Pix2pix (CVPR’17): Pairwise cross-domain training data• CycleGAN/DualGAN/DiscoGAN: Unpaired cross-domain training data• UNIT (NIPS’17): Learning cross-domain image representation (with unpaired training data)• DTN (ICLR’17) : Learning cross-domain image representation (with unpaired training data)

• Joint Image Translation & Disentanglement• StarGAN (CVPR’18) : Image translation via representation disentanglement• CDRD (CVPR’18) : Cross-domain representation disentanglement and translation

From Image Understanding to Image Manipulation

Pix2pix

• Image-to-image translation with conditional adversarial networks (CVPR’17)• Can be viewed as image style transfer

Sketch Photo

Isola et al. " Image-to-image translation with conditional adversarial networks." CVPR 2017. 9

Pix2pix• Goal / Problem Setting

• Image translation across two distinct domains (e.g., sketch v.s. photo)

• Pairwise training data

• Method: Conditional GAN• Example: Sketch to Photo

• Generator (VAE)Input: SketchOutput: Photo

• DiscriminatorInput: Concatenation of Input(Sketch) & Synthesized/Real(Photo) imagesOutput: Real or Fake

Testing Phase

GeneratedInput

Concat

Training Phase

Pix2pix

• Learning the model

GeneratedInput

Concat

Training Phase

Concat

ℒ𝑐𝑐𝐺𝐺𝐺𝐺𝐺𝐺(G, D) = 𝔼𝔼 𝑥𝑥 log 1 − D(𝑥𝑥, G(𝑥𝑥)) + 𝔼𝔼 𝑥𝑥,𝑦𝑦 log D 𝑥𝑥,𝑦𝑦Fake (Generated) Real

Concatenate Concatenate

ℒ𝐿𝐿𝐿(G) = 𝔼𝔼 𝑥𝑥,𝑦𝑦 𝑦𝑦 − G(𝑥𝑥) 𝐿

Reconstruction Loss

Conditional GAN loss

Overall objective functionG∗ = arg min

ℒ𝑐𝑐𝐺𝐺𝐺𝐺𝐺𝐺 G, D + ℒ𝐿𝐿𝐿(G)

Pix2pix

• Experiment results

Demo page: https://affinelayer.com/pixsrv/

CycleGAN/DiscoGAN/DualGAN

• CycleGAN (CVPR’17)• Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial

Networks -to-image translation with conditional adversarial networks

Zhu et al. "Unpaired Image-To-Image Translation Using Cycle-Consistent Adversarial Networks." CVPR 2017.

• Easier to collect training data

• More practical

Paired Unpaired

1-to-1 Correspondence

No Correspondence

CycleGAN

• Goal / Problem Setting• Image translation across two distinct domains • Unpaired training data

• Idea• Autoencoding-like image translation• Cycle consistency between two domains

Photo PaintingUnpaired

Cycle Consistency

Photo Painting Photo

Training data

Painting Photo Painting

Cycle Consistency

CycleGAN

• Method (Example: Photo & Painting)

• Based on 2 GANs• First GAN (G1, D1): Photo to Painting• Second GAN (G2, D2): Painting to Photo

• Cycle Consistency• Photo consistency• Painting consistency

Photo(Input)

Painting(Generated)

Painting(Real)

or Real / Fake

Photo(Generated)

Painting(Input)

D2or Real / Fake

Photo(Real)

CycleGAN

• Method (Example: Photo vs. Painting)

• Based on 2 GANs• First GAN (G1, D1): Photo to Painting• Second GAN (G2, D2): Photo to Painting

• Cycle Consistency• Photo consistency• Painting consistency

Photo Consistency

Photo Painting PhotoG1 G2

Painting Consistency

CycleGAN

• Learning

• Adversarial Loss• First GAN (G1, D1):

• Second GAN (G2, D2):

Overall objective functionG𝐿∗ , G2∗ = arg min

G1,G2maxD1,D2

ℒ𝐺𝐺𝐺𝐺𝐺𝐺 G𝐿, D𝐿 + ℒ𝐺𝐺𝐺𝐺𝐺𝐺 G2, D2 + ℒ𝑐𝑐𝑦𝑦𝑐𝑐 G𝐿, G2First GAN Second GAN

ℒ𝐺𝐺𝐺𝐺𝐺𝐺 G𝐿, D𝐿 = 𝔼𝔼 log 1 − D𝐿(G𝐿(𝑥𝑥)) + 𝔼𝔼 log D𝐿 𝑦𝑦

ℒ𝐺𝐺𝐺𝐺𝐺𝐺 G2, D2 = 𝔼𝔼 log 1 − D2(G2(𝑦𝑦)) + 𝔼𝔼 log D2 𝑥𝑥

Photo(Input)

Painting(Generated)

Painting(Real)

Real/ Fake

Photo(Generated)

Painting(Input)

D2or Real/ Fake

Photo(Real)

𝑥𝑥 G𝐿(𝑥𝑥)

𝑦𝑦

𝑦𝑦 G2(𝑦𝑦)

𝑥𝑥

CycleGAN

• Learning

• Consistency Loss• Photo and Painting consistency

Overall objective functionG𝐿∗ , G2∗ = arg min

G1,G2maxD1,D2

ℒ𝐺𝐺𝐺𝐺𝐺𝐺 G𝐿, D𝐿 + ℒ𝐺𝐺𝐺𝐺𝐺𝐺 G2, D2 + ℒ𝑐𝑐𝑦𝑦𝑐𝑐 G𝐿, G2Cycle Consistency

ℒ𝑐𝑐𝑦𝑦𝑐𝑐 G𝐿, G2 = 𝔼𝔼 G2 G𝐿 𝑥𝑥 − 𝑥𝑥 𝐿 + G𝐿 G2 𝑦𝑦 − 𝑦𝑦 𝐿

Photo Consistency

Photo Painting Photo

G1 G2𝑥𝑥 G𝐿 𝑥𝑥 G2 G𝐿 𝑥𝑥

Painting Consistency

G1G2𝑦𝑦 G2 𝑦𝑦 G𝐿 G2 𝑦𝑦

CycleGAN

• Example results

Project Page: https://junyanz.github.io/CycleGAN/

Image Translation Using Unpaired Training Data

Zhu et al. "Unpaired Image-To-Image Translation Using Cycle-Consistent Adversarial Networks." CVPR 2017.Kim et al. "Learning to Discover Cross-Domain Relations with Generative Adversarial Networks.“, ICML 2017

Yi et al. "DualGAN: Unsupervised dual learning for image-to-image translation." ICCV 2017

• CycleGAN, DiscoGAN, and DualGAN

CycleGANICCV’17

DiscoGANICML’17

DualGANICCV’17

• Unsupervised Image-to-Image Translation Networks (NIPS’17)• Image translation via learning cross-domain joint representation

Liu et al., "Unsupervised image-to-image translation networks.“, NIPS 2017

𝑥𝑥𝐿𝑥𝑥2

𝑧𝑧𝒵𝒵: Joint latent space

𝒳𝒳𝐿 𝒳𝒳2

𝑥𝑥𝐿𝑥𝑥2

𝑧𝑧𝒵𝒵: Joint latent space

𝒳𝒳𝐿 𝒳𝒳2

Stage1: Encode to the joint space Stage2: Generate cross-domain images

Day Night Day Night

• Goal/Problem Setting• Image translation

across two distinct domains• Unpaired training image data

• Idea• Based on two parallel VAE-GAN models

VAE GAN

• Idea• Based on two parallel VAE-GAN models• Learning of joint representation

across image domains

• Idea• Based on two parallel VAE-GAN models• Learning of joint representation

across image domains• Generate cross-domain images

from joint representation

Variation Autoencoder Loss

• LearningOverall objective function

G∗ = arg minG

ℒ𝑉𝑉𝐺𝐺𝑉𝑉 E𝐿, G𝐿, E2, G2 + ℒ𝐺𝐺𝐺𝐺𝐺𝐺 G𝐿, D𝐿, G2, D2

ℒ𝑉𝑉𝐺𝐺𝑉𝑉 E𝐿, G𝐿, E2, G2 = 𝔼𝔼 G𝐿 E𝐿 𝑥𝑥𝐿 − 𝑥𝑥𝐿 2 + 𝔼𝔼 𝒦𝒦ℒ(𝑞𝑞𝐿(𝑧𝑧)||𝑝𝑝(𝑧𝑧))𝔼𝔼 G2 E2 𝑥𝑥2 − 𝑥𝑥2 2 + 𝔼𝔼 𝒦𝒦ℒ(𝑞𝑞2(𝑧𝑧)||𝑝𝑝(𝑧𝑧))

VAEG𝐿E𝐿 D𝐿

E2 G2 D2

Adversarial Lossℒ𝐺𝐺𝐺𝐺𝐺𝐺 G𝐿, D𝐿, G2, D2 = 𝔼𝔼 log 1 − D𝐿(G𝐿(𝑧𝑧) + 𝔼𝔼 log D𝐿 𝑦𝑦𝐿

𝔼𝔼 log 1 − D2(G2(𝑧𝑧) + 𝔼𝔼 log D2 𝑦𝑦2

G𝐿(𝑧𝑧)

G2(𝑧𝑧)

Variation Autoencoder Adversarial

ℒ𝐺𝐺𝐺𝐺𝐺𝐺 G𝐿, D𝐿, G2, D2 = 𝔼𝔼 log 1 − D𝐿(G𝐿(𝑧𝑧) + 𝔼𝔼 log D𝐿 𝑦𝑦𝐿𝔼𝔼 log 1 − D2(G2(𝑧𝑧) + 𝔼𝔼 log D2 𝑦𝑦2

G = arg minG

ℒ𝑉𝑉𝐺𝐺𝑉𝑉 E𝐿, G𝐿, E2, G2 = 𝔼𝔼 G𝐿 E𝐿 𝑥𝑥𝐿 − 𝑥𝑥𝐿 2 + 𝔼𝔼 𝒦𝒦ℒ(𝑞𝑞𝐿(𝑧𝑧)||𝑝𝑝(𝑧𝑧))

𝔼𝔼 G2 E2 𝑥𝑥2 − 𝑥𝑥2 2 + 𝔼𝔼 𝒦𝒦ℒ(𝑞𝑞2(𝑧𝑧)||𝑝𝑝(𝑧𝑧))

VAEG𝐿E𝐿 D𝐿

E2 G2 D2

Adversarial Loss

G𝐿 E𝐿 𝑥𝑥𝐿

G2 E2 𝑥𝑥2

Reconstruction

ℒ𝐺𝐺𝐺𝐺𝐺𝐺 G𝐿, D𝐿, G2, D2 = 𝔼𝔼 log 1 − D𝐿(G𝐿(𝑧𝑧) + 𝔼𝔼 log D𝐿 𝑦𝑦𝐿𝔼𝔼 log 1 − D2(G2(𝑧𝑧) + 𝔼𝔼 log D2 𝑦𝑦2

G = arg minG

ℒ𝑉𝑉𝐺𝐺𝑉𝑉 E𝐿, G𝐿, E2, G2 = 𝔼𝔼 G𝐿 E𝐿 𝑥𝑥𝐿 − 𝑥𝑥𝐿 2 + 𝔼𝔼 𝒦𝒦ℒ(𝑞𝑞𝐿(𝑧𝑧)||𝑝𝑝(𝑧𝑧))

𝔼𝔼 G2 E2 𝑥𝑥2 − 𝑥𝑥2 2 + 𝔼𝔼 𝒦𝒦ℒ(𝑞𝑞2(𝑧𝑧)||𝑝𝑝(𝑧𝑧))

VAEG𝐿E𝐿 D𝐿

E2 G2 D2

Adversarial Loss

G𝐿 E𝐿 𝑥𝑥𝐿

G2 E2 𝑥𝑥2

Prior Loss

G = arg minG

G𝐿E𝐿 D𝐿

E2 G2 D2

𝔼𝔼 log 1 − D2(G2(𝑧𝑧) + 𝔼𝔼 log D2 𝑦𝑦2

G𝐿(𝑧𝑧)

G2(𝑧𝑧)

Generated

𝑦𝑦𝐿

𝑦𝑦2

G = arg minG

𝔼𝔼 log 1 − D2(G2(𝑧𝑧) + 𝔼𝔼 log D2 𝑦𝑦2Real

G𝐿 D𝐿

G𝐿(𝑧𝑧)

G2(𝑧𝑧)

GAN𝑦𝑦𝐿

𝑦𝑦2Real

UNIT• Example results

Sunny → Rainy

Rainy → Sunny

Real Street-view → Synthetic Street-view

Synthetic Street-view → Real Street-view

Github Page: https://github.com/mingyuliutw/UNIT

Domain Transfer Networks

• Unsupervised Cross-Domain Image Generation (ICLR’17)• Goal/Problem Setting

• Image translation across two domains• One-way only translation• Unpaired training data

• Idea • Apply unified model to learn

joint representation across domains.

Taigman et al., "Unsupervised cross-domain image generation.“, ICLR 2016

• Unsupervised Cross-Domain Image Generation (ICLR’17)• Goal/Problem Setting

• Image translation across two domains• One-way only translation• Unpaired training data

• Idea • Apply unified model to learn

joint representation across domains.• Consistency observed in image and feature spaces

Image consistency

feature consistency

• Learning• Unified model to translate across domains

• Consistency of feature and image space

• Adversarial loss

G∗ = arg minG

ℒ𝑖𝑖𝑖𝑖𝑖𝑖 G + ℒ𝑓𝑓𝑓𝑓𝑓𝑓𝑓𝑓 G + ℒ𝐺𝐺𝐺𝐺𝐺𝐺 G, D

ℒ𝑖𝑖𝑖𝑖𝑖𝑖 G = 𝔼𝔼 𝑔𝑔 𝑓𝑓 𝑦𝑦 − 𝑦𝑦 2

ℒ𝑓𝑓𝑓𝑓𝑓𝑓𝑓𝑓 G = 𝔼𝔼 𝑓𝑓(𝑔𝑔 𝑓𝑓 𝑥𝑥 ) − 𝑓𝑓(𝑥𝑥) 2

ℒ𝐺𝐺𝐺𝐺𝐺𝐺 G, D = 𝔼𝔼 log 1 − D(G(𝑥𝑥) + 𝔼𝔼 log 1 − D(G(𝑦𝑦) + 𝔼𝔼 log D 𝑦𝑦

• Consistency of image and feature space

G∗ = arg minG

𝑦𝑦

𝑥𝑥

Image consistency

feature consistency

G = {𝑓𝑓,𝑔𝑔}

G∗ = arg minG

𝑦𝑦

𝑥𝑥

G DG(𝑦𝑦)

G(𝑥𝑥)

G∗ = arg minG

𝑦𝑦

𝑥𝑥

G DG(𝑦𝑦)

G(𝑥𝑥)

DTN• Example results

SVHN 2 MNIST Photo 2 Emoji

StarGAN

• Goal• Unified GAN for multi-domain image-to-image translation

Choi et al. "StarGAN: Unified Generative Adversarial Networks for Multi-Domain Image-to-Image Translation." CVPR 2018

Traditional Cross-Domain Models Unified Multi-Domain Model(StarGAN)

StarGAN

• Goal• Unified GAN for multi-domain image-to-image translation

Traditional Cross-Domain ModelsUnified Multi-Domain Model

(StarGAN)

UnifiedG

𝐺𝐺𝐿2

𝐺𝐺𝐿3 𝐺𝐺24

𝐺𝐺34

𝐺𝐺𝐿4

𝐺𝐺23

StarGAN

• Goal / Problem Setting• Single image translation model across

multiple domains • Unpaired training data

StarGAN

• Goal / Problem Setting• Single Image translation model across multiple

domains • Unpaired training data

• Idea• Concatenate image and target domain label as input of generator• Auxiliary domain classifier on Discriminator

Target domain Image

StarGAN

• Goal / Problem Setting• Single Image translation model across multiple

domains • Unpaired training data

• Idea• Concatenate image and target domain label as input of

Generator• Auxiliary domain classifier as discriminator too

StarGAN

• Goal / Problem Setting• Single Image translation model across

multiple domains • Unpaired training data

• Idea• Concatenate image and target domain label as input of

Generator• Auxiliary domain classifier on Discriminator• Cycle consistency across domains

StarGAN• Goal / Problem Setting

• Single Image translation model across multiple domains

• Unpaired training data

• Idea• Auxiliary domain classifier as discriminator• Concatenate image and target domain label as input• Cycle consistency across domains

StarGAN

G∗ = arg minG

ℒ𝐺𝐺𝐺𝐺𝐺𝐺 G, D + ℒ𝑐𝑐𝑐𝑐𝑐𝑐 G, D + ℒ𝑐𝑐𝑦𝑦𝑐𝑐 G

StarGAN

• Learning

ℒ𝐺𝐺𝐺𝐺𝐺𝐺 G, D = 𝔼𝔼 log 1 − D(G(𝑥𝑥, 𝑐𝑐)) + 𝔼𝔼 log D 𝑦𝑦

Adversarial Loss 𝑦𝑦G(𝑥𝑥, 𝑐𝑐)

𝑥𝑥𝑐𝑐

StarGAN

• Learning

• Domain Classification Loss (Disentanglement)

Domain Classification Loss 𝑦𝑦G(𝑥𝑥, 𝑐𝑐)

𝑥𝑥𝑐𝑐

StarGAN

• Learning

Real dataw.r.t. its domain label

𝑐𝑐

𝑥𝑥

Dcls(𝑐𝑐′|𝑦𝑦)

StarGAN

• Learning

𝑐𝑐 𝑥𝑥

Dcls(𝑐𝑐|G(𝑥𝑥, 𝑐𝑐))

StarGAN

• Learning

• Cycle Consistency Loss

Consistency Loss

𝑐𝑐𝑥𝑥

G(𝑥𝑥, 𝑐𝑐)

𝑥𝑥𝑐𝑐ℒ𝑐𝑐𝑐𝑐𝑐𝑐 G, D= 𝔼𝔼 − log Dcls(𝑐𝑐′|𝑦𝑦) + 𝔼𝔼 − log Dcls(𝑐𝑐|G(𝑥𝑥, 𝑐𝑐))

ℒ𝑐𝑐𝑦𝑦𝑐𝑐 G = 𝔼𝔼 G G 𝑥𝑥, 𝑐𝑐 , 𝑐𝑐𝑥𝑥 − 𝑥𝑥 𝐿

G G 𝑥𝑥, 𝑐𝑐 , 𝑐𝑐𝑥𝑥

StarGAN

• Learning

• Domain Classification Loss

• Cycle Consistency Loss

ℒ𝑐𝑐𝑦𝑦𝑐𝑐 G = 𝔼𝔼 G G 𝑥𝑥, 𝑐𝑐 , 𝑐𝑐𝑥𝑥 − 𝑥𝑥 𝐿

StarGAN• Example results

• StarGAN can somehow be viewed as a representation disentanglement model, instead of an image translation one.

Multiple Domains Multiple Domains

Github Page: https://github.com/yunjey/StarGAN

Cross-Domain Representation Disnentanglement (CDRD)

Wang et al., Detach and Adapt: Learning Cross-Domain Disentangled Deep Representation”. CVPR 2018

• Goal / Problem Setting• Learning cross-domain joint disentangled

representation • Single domain supervision• Cross-domain image translation with attribute of

interest

• Idea• Bridge the domain gap across domains• Auxiliary attribute classifier on Discriminator• Semantics of disentangled factor is learned from label

in source domain

Joint Space

• Goal / Problem Setting• Learning cross-domain joint disentangled representation • Single domain supervision• Cross-domain image translation with attribute of interest

• Idea• Bridge the domain gap across domains• Auxiliary attribute classifier on Discriminator• Semantics of disentangled factor is learned from label in

source domain

Supervised(w/ or w/oEyeglass)

Unsupervised

w/ eyeglasses

w/o eyeglasses

• Idea• Bridge the domain gap across domains• Auxiliary attribute classifier on Discriminator• Semantics of disentangled factor is learned from label in

source domain

w/ eyeglasses

w/o eyeglasses

• Idea• Based on GAN• Auxiliary attribute classifier on Discriminator• Bridge the domain gap across domains• Semantics of disentangled factor is learned from label in

source domain

Real or Fake

𝑧𝑧

Generator

Discriminator

𝑋𝑋𝑆𝑆 �𝑋𝑋𝑆𝑆

• Idea• Based on GAN• Auxiliary attribute classifier as Discriminator• Bridge the domain gap across domains• Semantics of disentangled factor is learned from label in

source domain

Real or Fake

𝑧𝑧

Generator

Discriminator

𝑋𝑋𝑆𝑆 �𝑋𝑋𝑆𝑆

𝑙𝑙

Classificationw.r.t. 𝑙𝑙𝑆𝑆 or 𝑙𝑙

𝑙𝑙𝑆𝑆

Supervised

• Idea• Based on GAN• Auxiliary attribute classifier on Discriminator• Bridge the domain gap

with division of high and low-level layers• Semantics of disentangled factor is learned from label in

source domain

• Idea• Based on GAN• Auxiliary attribute classifier on Discriminator• Bridge the domain gap with division of high- and low-

level layer• Semantics of disentangled factor is learned from

label info in source domain

G∗ = arg minG

ℒ𝐺𝐺𝐺𝐺𝐺𝐺𝑆𝑆 G, D = 𝔼𝔼 log D𝐶𝐶(DS(𝑋𝑋𝑆𝑆)) + 𝔼𝔼 log 1 − D𝐶𝐶(DS( �𝑋𝑋𝑆𝑆))

ℒ𝐺𝐺𝐺𝐺𝐺𝐺𝑇𝑇 G, D = 𝔼𝔼 log D𝐶𝐶(DT(𝑋𝑋𝑇𝑇)) + 𝔼𝔼 log 1 − D𝐶𝐶(DT( �𝑋𝑋𝑇𝑇))

ℒ𝐺𝐺𝐺𝐺𝐺𝐺 G, D = ℒ𝐺𝐺𝐺𝐺𝐺𝐺𝑆𝑆 G, D + ℒ𝐺𝐺𝐺𝐺𝐺𝐺𝑇𝑇 G, D

Adversarial Loss

G∗ = arg minG

ℒ𝐺𝐺𝐺𝐺𝐺𝐺 G, D = ℒ𝐺𝐺𝐺𝐺𝐺𝐺𝑆𝑆 G, D + ℒ𝐺𝐺𝐺𝐺𝐺𝐺𝑇𝑇 G, DAdversarial Loss

Generated

G∗ = arg minG

AC-GAN

G∗ = arg minG

ℒ𝐺𝐺𝐺𝐺𝐺𝐺𝑆𝑆 G, D = 𝔼𝔼 log D𝐶𝐶(DS(𝑋𝑋𝑆𝑆)) + 𝔼𝔼 log 1 − D𝐶𝐶(DS( �𝑋𝑋𝑆𝑆))ℒ𝐺𝐺𝐺𝐺𝐺𝐺𝑇𝑇 G, D = 𝔼𝔼 log D𝐶𝐶(DT(𝑋𝑋𝑇𝑇)) + 𝔼𝔼 log 1 − D𝐶𝐶(DT( �𝑋𝑋𝑇𝑇))

ℒ𝑐𝑐𝑐𝑐𝑐𝑐𝑆𝑆 G, D = 𝔼𝔼 log𝑃𝑃(𝑙𝑙 = 𝑙𝑙| �𝑋𝑋𝑆𝑆) + 𝔼𝔼 log𝑃𝑃(𝑙𝑙 = 𝑙𝑙𝑆𝑆|𝑋𝑋𝑆𝑆)

ℒ𝑐𝑐𝑐𝑐𝑐𝑐𝑇𝑇 G, D = 𝔼𝔼 log𝑃𝑃(𝑙𝑙 = 𝑙𝑙| �𝑋𝑋𝑇𝑇)

Disentangle Lossℒ𝑐𝑐𝑐𝑐𝑐𝑐 G, D = ℒ𝑐𝑐𝑐𝑐𝑐𝑐𝑆𝑆 G, D + ℒ𝑐𝑐𝑐𝑐𝑐𝑐𝑇𝑇 G, D

InfoGAN

G∗ = arg minG

ℒ𝐺𝐺𝐺𝐺𝐺𝐺𝑆𝑆 G, D = 𝔼𝔼 log D𝐶𝐶(DS(𝑋𝑋𝑆𝑆)) + 𝔼𝔼 log 1 − D𝐶𝐶(DS( �𝑋𝑋𝑆𝑆))ℒ𝐺𝐺𝐺𝐺𝐺𝐺𝑇𝑇 G, D = 𝔼𝔼 log D𝐶𝐶(DT(𝑋𝑋𝑇𝑇)) + 𝔼𝔼 log 1 − D𝐶𝐶(DT( �𝑋𝑋𝑇𝑇))

ℒ𝑐𝑐𝑐𝑐𝑐𝑐𝑆𝑆 G, D = 𝔼𝔼 log𝑃𝑃(𝑙𝑙 = 𝑙𝑙| �𝑋𝑋𝑆𝑆) + 𝔼𝔼 log𝑃𝑃(𝑙𝑙 = 𝑙𝑙𝑆𝑆|𝑋𝑋𝑆𝑆)

ℒ𝑐𝑐𝑐𝑐𝑐𝑐𝑇𝑇 G, D = 𝔼𝔼 log𝑃𝑃(𝑙𝑙 = 𝑙𝑙| �𝑋𝑋𝑇𝑇)

Disentangle Lossℒ𝑐𝑐𝑐𝑐𝑐𝑐 G, D = ℒ𝑐𝑐𝑐𝑐𝑐𝑐𝑆𝑆 G, D + ℒ𝑐𝑐𝑐𝑐𝑐𝑐𝑇𝑇 G, D

CDRD• Add an additional encoder

• Input: Gaussian Noise Image• Image translation with attribute of interest• Input: Gaussian Noise

• Experiment resultsNo Label Supervision

Wang et al., Detach and Adapt: Learning Cross-Domain Disentangled Deep Representation”. CVPR 2018 72

CDRD• Experiment results

Digits

Cross-Domain Classification

Figure: t-SNE for digit

CVPR’17

NIPS’16

ICML’15

ICCV’15

CVPR’17

PAMI’17

CVPR’14

CVPR’13

ECCV’16

arXiv’16

NIPS’16

NIPS’17

NIPS’16

NIPS’17

Wang et al., Detach and Adapt: Learning Cross-Domain Disentangled Deep Representation”. CVPR 2018 73

Comparisons

Cross-Domain Image Translation Representation Disentanglement

Unpaired Training Data

Multi-domains Bi-direction Joint

Representation Unsupervised Interpretability of disentangled factor

Pix2pix X X X X

Cannot disentangle representation

CycleGAN O X O XStarGAN O O O X

UNIT O X O ODTN O X X O

infoGAN

Cannot translate image across domainsO X

AC-GAN X OCDRD (Ours) O O O O Partially O

What’s to Be Covered Today…

• Visualization of NNs• t-SNE• Visualization of Feature Maps• Neural Style Transfer

Many slides from Fei-Fei Li, Bhiksha Raj, Yuying Yeh, and Hsuan-I Ho 75

What We Have Learned…

• CNN for Image Classification

MONKEY

What We Have Learned…

• Generative Models (e.g., GAN) for Image Synthesis

But…

• Is it sufficiently effective and robust to perform visual analysis from a single image?

• E.g., Are the hands of this man tied?

But…

• Is it sufficiently effective and robust to perform visual analysis from a single image?

• E.g., Are the hands of this man tied?

So What Are The Limitations of CNN?

• Can’t easily model series data• Both input and output might be sequential data (scalar or vector)

• Simply feed-forward processing• Cannot memorize, no long-term feedback

MONKEY

Example of (Visual) Sequential Data

https://quickdraw.withgoogle.com/#

More Applications in Vision

Image Captioning

Figure from Vinyals et al, “Show and tell: A neural image caption generator”, CVPR 201582

More Applications in Vision

Visual Question Answering (VQA)

output

Figure from Zhu et al, “Visual 7W: Grounded Question Answering in Images”, CVPR 201683

How to Model Sequential Data?

• Deep learning for sequential data• Recurrent neural networks (RNN)• 3-dimensional convolution neural networks

RNN 3D convolution

Recurrent Neural Networks

• Parameter sharing + unrolling• Keeps the number of parameters fixed• Allows sequential data with varying lengths

• Memory ability• Capture and preserve information which has been extracted

Recurrence Formula

• Same function and parameters used at every time step:

ℎ𝑓𝑓 ,𝑦𝑦𝑓𝑓 = 𝑓𝑓𝑊𝑊 ℎ𝑓𝑓−𝐿, 𝑥𝑥𝑓𝑓

𝑥𝑥

𝑦𝑦new statefor time t

state attime t-1

input vector at time t

output vectorat time t

functionwith parameters W

Recurrence Formula

• Same function and parameters used at every time step:

ℎ𝑓𝑓 ,𝑦𝑦𝑓𝑓 = 𝑓𝑓𝑊𝑊 ℎ𝑓𝑓−𝐿, 𝑥𝑥𝑓𝑓

ℎ𝑓𝑓 = 𝑡𝑡𝑡𝑡𝑡𝑡ℎ 𝑊𝑊ℎℎℎ𝑓𝑓−𝐿 + 𝑊𝑊𝑥𝑥ℎ𝑥𝑥𝑓𝑓𝑦𝑦𝑓𝑓 = 𝑊𝑊ℎ𝑦𝑦ℎ𝑓𝑓

𝑥𝑥

𝑦𝑦

Multiple Recurrent Layers

𝑥𝑥𝐿

𝑓𝑓𝑊𝑊 ℎ𝐿ℎ0

𝑥𝑥𝐿

𝑓𝑓𝑊𝑊 ℎ𝐿ℎ0 𝑓𝑓𝑊𝑊

𝑥𝑥2

𝑥𝑥𝐿

𝑥𝑥2

ℎ2……

ℎ𝑇𝑇𝑓𝑓𝑊𝑊

𝑥𝑥3

𝑥𝑥𝐿

𝑥𝑥2

ℎ2……

𝑥𝑥3

𝑥𝑥𝐿

𝑥𝑥2

ℎ2……

𝑥𝑥3

𝑦𝑦𝐿 𝑦𝑦2 𝑦𝑦3 𝑦𝑦𝑇𝑇

e.g., video indexinge.g., video predictione.g., image caption e.g., action recognition

Example: Image Captioning

Figure from Karpathy et a, “Deep Visual-Semantic Alignments for Generating Image Descriptions”, CVPR 201596

Example: Action Recognition

Figure from “Action Recognition in Video Sequences using Deep Bi-Directional LSTM With CNN Features” 104

𝑥𝑥𝐿

𝑥𝑥2

ℎ2……

𝑥𝑥3

𝑥𝑥𝐿

𝑥𝑥2

ℎ2……

𝑥𝑥3

softmax

standingexercise

𝑥𝑥𝐿

𝑥𝑥2

ℎ2……

𝑥𝑥3

action label

softmax

cross entropyloss

Back propagation through time

Back Propagation Through Time (BPTT)

Pascanu et al, “On the difficulty of training recurrent neural networks”, ICML 2013

Back Propagation Through Time (BPTT)

Training RNNs via BPTT

Training RNNs: Forward Pass

• Forward pass each training instance:• Pass the entire data sequence through the net• Generate outputs

Training RNNs: Computing Gradients

• After forward passing each training instance:• Backward pass: compute gradients via BP• More specifically, BPTT

Training RNNs: BPTT

• Let’s focus on one training instance.• The divergence computed is between

the sequence of outputs by the networkand the desired sequence of outputs.

• Generally, this is not just the sum of the divergences at individual times.

Gradient Vanishing & Exploding

● Computing gradient involves many factors of W

○ Exploding gradients : Largest singular value > 1

○ Vanishing gradients : Largest singular value < 1

Solutions…

● Gradients clipping : rescale gradients if too large

● How about vanishing gradients?

standard gradient descent trajectories

gradient clipping to fix problem

Variants of RNN

• Long Short-term Memory (LSTM) [Hochreiter et al., 1997]

• Additional memory cell • Input/Forget/Output Gates• Handle gradient vanishing• Learn long-term dependencies

• Gated Recurrent Unit (GRU) [Cho et al., EMNLP 2014]

• Similar to LSTM• No additional memory cell• Reset / Update Gates• Fewer parameters than LSTM• Comparable performance to LSTM [Chung et al., NIPS Workshop 2014]

Vanilla RNN vs. LSTM

Memory Cell

Input in time t

Output in time t

Forget gate

Input gate

Input activation

Output gate

Cell state

Hidden state

Long Short-Term Memory (LSTM)

Image Credit: Hung-Yi Lee 118

Memory Cell

Input in time tOutput in time t

Forget gate

Input gate

Input activation

Output gate

Cell state

Hidden state

Cell state

Hidden state

Forget gate

Input gate

Input activation

Output gate

Cell state

Hidden state

Calculate forget gate fCell state

Hidden state

Forget gate

Calculate forget gate fCell state

Hidden state

Input gate

Calculate input activationCell state

Hidden state

Input activation

Calculate output gateCell state

Hidden state

Output gate

Update memory cellCell state

Hidden state

Cell state

Calculate output htCell state

Hidden state

Prevent gradient vanishing if forget gate is open (>0).

Remarks:- Forget gate f provides shortcut connection across time steps- Linear relationship between ct and ct-1 instead of multiplication relationship of ht and ht-1 in vanilla RNN

Cell state

Hidden state

Variants of RNN

• Long Short-term Memory (LSTM) [Hochreiter et al., 1997]

• Additional memory cell • Input / Forget / Output Gates• Handle gradient vanishing• Learn long-term dependencies

• Gated Recurrent Unit (GRU) [Cho et al., EMNLP 2014]

• Similar to LSTM• No additional memory cell• Reset/Update Gates• Fewer parameters than LSTM• Comparable performance to LSTM [Chung et al., NIPS Workshop 2014]

LSTM vs. GRU

LSTM GRU

Forget gate

Input gate

Input activation

Output gate

Cell state

Hidden state

Reset gate

Update gate

Input activation

Hidden state

Gated Recurrent Unit (GRU)

Hidden state

Hidden stateCalculate reset gate r

Reset gate

Hidden stateCalculate update gate z

Update gate

Hidden stateCalculate input activation

Input activation

Hidden stateCalculate output

Hidden state

Hidden statePrevent gradient vanishing if update gate is open!

Remarks:- Update gate z provides shortcut connection across time steps- Linear relationship between ht and ht-1 instead of multiplication relationship of ht and ht-1 in vanilla RNN

Input in time tOutput in time t

Vanilla RNN, LSTM, & GRU

𝑥𝑥

𝑦𝑦

LSTM vs. GRU

Vanilla RNN LSTM GRU

Cell state

Number of Gates

Parameters

Gradient Vanishing / Exploding

References

bookhttp://www.deeplearningbook.org/contents/rnn.html

VThttps://computing.ece.vt.edu/~f15ece6504/slides/L14_RNNs.pptx.pdfhttps://computing.ece.vt.edu/~f15ece6504/slides/L15_LSTMs.pptx.pdf

CS231nhttp://cs231n.stanford.edu/slides/2017/cs231n_2017_lecture10.pdf

Jia-Bing Huanghttps://filebox.ece.vt.edu/~jbhuang/teaching/ece5554-4554/fa17/lectures/Lecture_23_ActionRec.pdf

MLDShttps://www.csie.ntu.edu.tw/~yvchen/f106-adl/doc/171030+171102_Attention.pdf

Web Tutorial

● http://www.wildml.com/2015/09/recurrent-neural-networks-tutorial-part-1-introduction-to-rnns/

● https://r2rt.com/written-memories-understanding-deriving-and-extending-the-lstm.html

What We’ve Learned Today…

• HW4 is due 5/18 Fri 23:59!

Deep Learning for Computer Visionvllab.ee.ntu.edu.tw/uploads/1/1/1/6/111696467/dlcv_w9__1...Chen et...

Documents

E learning b-learning-m-learning

e-learning e-learning, (e-learning),

CENTRA e-Learning & b-Learning

E- LEARNING Y B- LEARNING

Deep Learning for Computer Visionvllab.ee.ntu.edu.tw/uploads/1/1/1/6/111696467/dlcv_w13_part_ii.pdf · • Depth info can be estimated from the disparity map • Disparity map is

Learning from Learning Curves: Item Response Theory & Learning Factors Analysis

E-learning, b-learning, m-learning

VISION & LEARNING LABvllab.ee.ntu.edu.tw/uploads/1/1/1/6/111696467/dlcv_w16.pdfOutline for Few-Shot Learning • Introduction • Hallucination Approaches (learning to augment) •

U-learning tv-learning

Deep Learning for Computer Visionvllab.ee.ntu.edu.tw/uploads/1/1/1/6/111696467/dlcv_w7.pdfStochastic Neighbor Embedding (SNE) (cont’d) • Neighborhood Distributions • Consider

Learning Analytics: Open & Authentic Learning

E-learning & B-learning

Personal Learning Environments in Future Learning Scenarios€¦ · Learning scenario, personal learning environment, social networks. INTRODUCTION Presumably, learning scenarios

E learning, b-learning & m-learning

E-learning aula 1docente.ifsc.edu.br/joelma.kremer/MaterialDidatico/E-learning/E... · • Apresentar ferramentas de e-learning, m-learning, b-learning e u-learning aprendidas no

E learning-m-learning-b-learning

B learning-E-learning

Deep Learning for Computer Visionvllab.ee.ntu.edu.tw/uploads/1/1/1/6/111696467/dlcv_w1.pdf · TAs & Office Hours: TA Hours: as listed above, or by appointment TA email: ntudlcvta2019@gmail.com

E learning vs b-learning

E-LEARNING · "story-learning", "action-learning", "tutoriel" et "afest" e-learning digital-learning m-learning micro-learning blended-learning social-learning classe virtuelle adaptive