PowerPoint

, Dec 26, 2016 / Seoul National University

? ? ?Convolutional Neural Network CNN(ILSVRC winners)Tensorflow CNN CNN

#

Unsupervised LearningReinforcement LearningRNN

#

Nicolaus Copernicus(1473-1543)

#

?

?

? ?

#

1, 2, 3, 4 Steam EngineElectricity, AutomobilePC, InternetArtificial Intelligence

?1234

#

, , Credit : Nvidia blog

#

(Artificial Intelligence)1956 From Wikipedia:Artificial intelligence(AI) isintelligenceexhibited bymachines. Incomputer science, an ideal "intelligent" machine is a flexiblerational agentthat perceives its environment and takes actions that maximize its chance of success at some goal

AI Translation: (AI) . .

#

?

#

, . .

vsEasyHard

#

(?)

#

?

Credit : MBC

#

(Machine Learning)(Machine Learning)Machine learningis the subfield ofcomputer sciencethat "gives computers the ability to learn without being explicitly programmed " .

ComputerinputsprogramoutputsComputerinputsoutputsprogram programming program !

#

Quiz ?3 x + 2 x = 11 x + 4 x = -35 x + 5 x = 08 x + 3 x = 5 = 1, = -1

(3, 2), (1, 4), (5, 5), (8, 3) input data, 1, -3, 0, 5 label weight weight

#

Supervised Learning()Input labels (classification), (regression)Unsupervised Learning()Input (clustering), (compression)Reinforcement Learning()Label reward Action selection, policy learning

#

Reinforcement Learning

https://youtu.be/Q70ulPJW3Gk

https://youtu.be/SuoouILpjDo

#

#

Perceptron(Artificial Neural Network)

sigmoid activation function

#

Example of ANN(logical AND)

011

weight (-30, 20, 20) !

#

(Regression) ()() (km) 1 ()

#

data ) x 2 + x 0.3 + x (-1) + x 0.1 = ()

#

data ,,, . x + x + x + x = ()? ,,, random ()2 ,,, 1 , ( )2 = 287.524

,,, ()2 0 ,

#

Multi Layer Perceptron(MLP)

? (regression) input (classification) perceptron Linear fitting Non-linear transform

#

(Deep Learning)

(Deep Learning) deep neural network Hidden layer = 2 deep network

weight ??

#

Back Propagation

gradient descent

#

Vanishing gradient problemOverfitting problemGet stuck in local minima

#

Vanishing Gradient ProblemGradient Sigmoid ( : ) layer

#

Overfitting ProblemData data , data(test data)

#

Local Minima local minima

#

?Vanishing gradient problem Sigmoid ReLU Overfitting problem Regularization method ( : dropout)Get stuck in local minima Local minima

#

ReLU : Rectified Linear UnitReLU activation function sparse activationReLU 0 1 vanishing gradient

#

ReLU Piece-wise linear tiling : locally linear mapping

#

Dropout(Regularization Method) , ( : 50%) random hidden layer unit Ensemble model model model

#

Local Minima minimum gradient DNN local minima saddle point local minima global minimum (neural network )

saddle point

#

ConvolutionalNeuralNetwork

Convolutional Neural Network convolution layer, pooling layer, fully-connected layer Parameter(weight) sharingConvolution pooling layer feature fully-connected layer class Picture Credit : The Data Science blog

#

CNN

tile , network tile feature (: )Newtork tile feature ( weight ) feature(: ) network tile network feature

#

Convolution1110001110001110011001100101010101

434243234=convolution1110001110001110011001100101010101

4

=convolutionfilterfeature mapInput or feature mapfilterfeature mapInput or feature map

Convolution : 1x1 + 1x0 + 1x1 + 0x0 + 1x1 + 1x0 + 0x1 + 0x0 + 1x1 = 4Filter

#

Convolution

Credit : Leonardos gitbook

#

Feature Extractor

Credit : Adit Deshpandes blog

#

Feature Extractor

Credit : Adit Deshpandes blog

#

Convolution(Multi Channel, Many Filters)111101111000011011100110001101011000011000111111001110001110001110011001100-10001000-10-10-11-10-10101010101-10-101000-10-10-1100-101010-101011-110-1-1310301-202023

=convolutionInput channel : 3Output channel : 2# of filters : 2

#

Convolution(Multi Channel, Many Filters)111101111000011011100110001101011000011000111111001110001110001110011001100-10001000-10-10-11-10-10101010101301-2020231-110-1-1310-10-1010-10-1010-1-100101010-10101Input channel : 3Output channel : 2# of filters : 2convolutionconvolutionconvolutionconvolutionconvolutionconvolution

#

Visualization of a Convolution Layer

Picture Credit : fundamentals of deep learning by Nikhil

#

ReLU301-2020231-110-1-1310

301002023

101000310ReLUReLU

#

Pooling LayerMax pooling

#

2x2 Max Pooling with Stride=130100202310100031032231131max poolingmax pooling

#

Fully-Connected Layer322311313223113121softmax0.80.2CatDog

#

Convolutional Neural Network layer , , feature , layer feature

#

Large Scale Image ClassificationImageNetOver 15ImageNetOver 15 million labeled high-resolution imagesRoughly 22,000 categoriesCollected from the webLabeled by human labelers using Amazons Mechanical Turk crowd-sourcing toolImageNet Large-Scale Visual Recognition Challenge (ILSVRC)Uses a subset of ImageNet1,000 categories1.2 million training images50,000 validation images150,000 test imagesReport two error rates:Top-1 and top-5

#

ImageNet Classification Results

Krizhevsky et al. 16.4% error(top-5)Next best (non-convnet) 26.2% error

All rankers use deep learning(Convnet)

Revolution of Depth!AlexNet

#

AlexNet(2012 winner)7 hidden layers, 650,000 neurons, 60M parameters2 GPU 1

ImageNet Classification with Deep Convolutional Neural Networks

#

AlexNet Result

ImageNet Classification with Deep Convolutional Neural Networks

#

GoogLeNet(2014 winner)Inception modul 1x1 convolution channel Fully connected layer global average pooling 5M parameters, 1.5B operations/evaluation

Inception moduleGoing Deeper With Convolutions

#

VGG(2nd place in 2014)3x3 filter Why??Convolution filter stack receptive field 2 3x3 filter = 5x5 filter3 3x3 filter = 7x7 filterParameter filter regularization

Very Deep Convolutional Networks for Large-Scale Image Recognition

#

Inception-v3Factorization of filters

Rethinking the Inception Architecture for Computer Vision

#

Residual Net(2015 winner)Revolution of DepthDeep Residual Learning for Image Recognition

#

ResNetLayer ?

56 layer 20 layer training error

Deep Residual Learning for Image Recognition

#

ResNet deep model training error Shallow model parameter deeper model copyExtra layers identity function Deep model optimization (identity )

A shallower model(18 layers)A deeper model(34 layers) Deep Residual Learning for Image Recognition

#

Deep Residual LearningIdentity layer ,

Deep Residual Learning for Image Recognition

#

ResNet ResultImageNet experiments

Deep Residual Learning for Image Recognition

#

Inception-ResNetInception + ResNet

Inception-v4, Inception-ResNet and the Impact of Residual Connections on Learning

#

ILSVRC 2016 Result

#

Tensorflow TensorflowGoogle , 2015 11 open source frameworkPython, computational graph (theano )CPU, GPU, multi-GPU https://tensorflow.org

#

Tensorflow : variable, placeholder a = tf.placeholder(float) , : network y = tf.mul(a, b)Insert coin : session sess = tf.Session() : network sess.run(y, feed_dict={a: 3, b: 3}

#

Global Average PoolingClass Activation MapGlobal average pooling , parameter (CNN parameter fc layer )Class activation map , class , class

Learning Deep Features for Discriminative Localization

#

Fully Connected Layer322311313223113121softmax0.80.2CatDog

#

Global Average Pooling

322311312.51.521softmax0.80.2CatDog

averageaverage

#

PPMI Datasethttp://ai.stanford.edu/~bangpeng/ppmi.html

#

CNN

Detection

SSD: Single Shot MultiBox Detector

#

Segmentation

Learning Deconvolution Network for Semantic Segmentation

#

Super Resolution

Deeply-Recursive Convolutional Network for Image Super-Resolution

#

Texture Synthesis

Texture Synthesis Using Convolutional Neural Networks

#

Artistic Style Transfer

Image Style Transfer Using Convolutional Neural Networks

#

Image Captioning

#

Visual QnA

Q: What is the boy holding?DPPnet:surfboardDPPnet:bat

Q: What is the animal doing?DPPnet:resting (relaxing)DPPnet:swimming (fishing)Image Question Answering using Convolutional Neural Network with Dynamic Parameter Prediction

#

Generative Adversarial Network

StackGAN: Text to Photo-realistic Image Synthesis with Stacked Generative Adversarial Networks

#

Autonomous Driving

#

Summary & Conclusion , () () , network data

#

, ? ? 1 20x20 ? data , ? ,

#

QnAThank You

jwlee@capp.snu.ac.kr

#