Embed Size (px)
Citation preview
1Terry Taewoong Um ([email protected])
University of Waterloo
Department of Electrical & Computer Engineering
Terry Taewoong Um
INTRODUCTION TO DEEP NEURAL NETWORK WITH TENSORFLOW
3Terry Taewoong Um ([email protected])
EXAMPLE CASE
- Imagine you have extracted features from sensors
- The dimension of each sample (which represents one of gestures) is around 800
- You have 70,000 samples (trial)
- What method would you apply?
4Terry Taewoong Um ([email protected])
EXAMPLE CASE
- Reduce the dimension from 800 to 40 by using a feature selection or dim. reduction technique
☞ What you did here is “Finding a good representation”
- Then, you may apply a classification methods to classify 10 classes
• You may have several ways to do it
• But, what if
- You have no idea for feature selection?
- The dimension is much higher than 800 and you have more classes.
?
5Terry Taewoong Um ([email protected])
EXAMPLE CASE
- Reduce the dimension from 800 to 40 by using a feature selection or dim. reduction technique
☞ What you did here is “Finding a good representation”
- Then, you may apply a classification methods to classify 10 classes
• You may have several ways to do it
• But, what if
- You have no idea for feature selection?
- The dimension is much higher than 800 and you have more classes.
MNIST dataset(65000spls * 784dim)
MNIST dataset(60000spls * 1024dim)
6Terry Taewoong Um ([email protected])
CLASSIFICATION RESULTS
error rate : 28% → 15% → 8%(2010) (2014)(2012)
http://rodrigob.github.io/are_we_there_yet/build/classification_datasets_results.html
7Terry Taewoong Um ([email protected])
PARADIGM CHANGE
Knowledge
PRESENT
Representation(Features)
How can we find a good representation?
IMAGE
SPEECH
Hand-Crafted Features
8Terry Taewoong Um ([email protected])
PARADIGM CHANGE
IMAGE
SPEECH
Hand-Crafted Features
Knowledge
PRESENT
Representation(Features)
Can we learn a good representation (feature) for the target task as well?
9Terry Taewoong Um ([email protected])
UNSUPERVISED LEARNING
“Convolutional deep belief networks for scalable unsupervised learning of hierarchical representation”, Lee et al., 2012
10Terry Taewoong Um ([email protected])
THREE TYPES OF DEEP LEARNING• Unsupervised learning method
Autoencoder http://goo.gl/s6kmqY
- Restricted Boltzmann Machine(RBM), Autoencoder, etc.
- It helps to avoid local minima problem (It regularizes the training data)
- But it is not necessary when we have large amount of data. (Drop-out is enough for regularization)
• Convolutional Neural Network (ConvNet)
• Recurrent Neural Network (RNN) + Long-Short Term Memory (LSTM)
- ConvNet has shown outstanding performance in recognition tasks (image, speech)
- ConvNet contains hierarchical abstraction process called pooling.
- RNN+LSTM makes use of long-term memory → Good for time-series data
- RNN is a generative model: It can generate new data
11Terry Taewoong Um ([email protected])
CONTENTS
2. DNN with TensorFlowThanks to Sungjoon Choi
https://github.com/sjchoi86/
15Terry Taewoong Um ([email protected])
BASIC WORKFLOW OF TF
1. Load data
2. Define the NN structure
3. Set optimization parameters
4. Run!https://github.com/terryum/TensorFlow_Exercises
17Terry Taewoong Um ([email protected])
1. LOAD DATA https://github.com/terryum/TensorFlow_Exercises/blob/master/2_LogisticRegression_MNIST_160516.ipynb
23Terry Taewoong Um ([email protected])
NEURAL NETWORK
Hugo Larochelle, http://www.dmi.usherb.ca/~larocheh/index_en.html
• Activation functions
http://goo.gl/qMQk5H
• Basic NN structure
24Terry Taewoong Um ([email protected])
1. LOAD DATA https://github.com/terryum/TensorFlow_Exercises/blob/master/3a_MLP_MNIST_160516.ipynb
29Terry Taewoong Um ([email protected])
CONVOLUTION
http://colah.github.io/posts/2014-07-Understanding-Convolutions/
http://www.wildml.com/2015/11/understanding-convolutional-neural-networks-for-nlp/
30Terry Taewoong Um ([email protected])
CONVOLUTIONAL NN
• How can we deal with real images which is much bigger than MNIST digit images?
- Use not fully-connected, but locally-connected NN
- Use convolutions to get various feature maps
- Abstract the results into higher layer by using pooling
- Fine tune with fully-connected NN
https://goo.gl/G7kBjI
https://goo.gl/Xswsbd
http://goo.gl/5OR5oH
31Terry Taewoong Um ([email protected])
1. LOAD DATA https://github.com/terryum/TensorFlow_Exercises/blob/master/4a_CNN_MNIST_160517.ipynb
37Terry Taewoong Um ([email protected])
Thank you
https://www.facebook.com/terryum
http://terryum.io/
http://t-robotics.blogspot.kr/
![Deep learning con TensorFloPor ejemplo, podr amos tener un tensor de tipo oat32 con rank 2 y con shape [3, 4]. Gabriel Munoz~ R os Deep learning con TensorFlow Variables Las variables](https://img.pdfslide.tips/doc/110x75/5e9964acb9507e4a325d4764/deep-learning-con-por-ejemplo-podr-amos-tener-un-tensor-de-tipo-oat32-con-rank.jpg){kind=icon}
![VINICIUSEIJIMARTINS€¦ · Ele utiliza uma framework de Deep Learning como base, podendo ela ser Tensorflow, Theano ouCNTK[9]. Ele possibilita a rápida construção e inferência](https://img.pdfslide.tips/doc/110x75/5f22742d6c81e0587531345d/vi-ele-utiliza-uma-framework-de-deep-learning-como-base-podendo-ela-ser-tensorflow.jpg)
