Introduction to Deep Learning with TensorFlow

1Terry Taewoong Um (terry.t.um@gmail.com)

University of Waterloo

Department of Electrical & Computer Engineering

Terry Taewoong Um

INTRODUCTION TO DEEP NEURAL NETWORK WITH TENSORFLOW

CONTENTS

1. Why Deep Neural Network

EXAMPLE CASE

- Imagine you have extracted features from sensors

- The dimension of each sample (which represents one of gestures) is around 800

- You have 70,000 samples (trial)

- What method would you apply?

EXAMPLE CASE

- Reduce the dimension from 800 to 40 by using a feature selection or dim. reduction technique

☞ What you did here is “Finding a good representation”

- Then, you may apply a classification methods to classify 10 classes

• You may have several ways to do it

• But, what if

- You have no idea for feature selection?

- The dimension is much higher than 800 and you have more classes.

EXAMPLE CASE

- Reduce the dimension from 800 to 40 by using a feature selection or dim. reduction technique

☞ What you did here is “Finding a good representation”

- Then, you may apply a classification methods to classify 10 classes

• You may have several ways to do it

• But, what if

- You have no idea for feature selection?

- The dimension is much higher than 800 and you have more classes.

MNIST dataset(65000spls * 784dim)

MNIST dataset(60000spls * 1024dim)

CLASSIFICATION RESULTS

error rate : 28% → 15% → 8%(2010) (2014)(2012)

http://rodrigob.github.io/are_we_there_yet/build/classification_datasets_results.html

PARADIGM CHANGE

Knowledge

PRESENT

Representation(Features)

How can we find a good representation?

SPEECH

Hand-Crafted Features

PARADIGM CHANGE

SPEECH

Hand-Crafted Features

Knowledge

PRESENT

Representation(Features)

Can we learn a good representation (feature) for the target task as well?

UNSUPERVISED LEARNING

“Convolutional deep belief networks for scalable unsupervised learning of hierarchical representation”, Lee et al., 2012

THREE TYPES OF DEEP LEARNING• Unsupervised learning method

Autoencoder http://goo.gl/s6kmqY

- Restricted Boltzmann Machine(RBM), Autoencoder, etc.

- It helps to avoid local minima problem (It regularizes the training data)

- But it is not necessary when we have large amount of data. (Drop-out is enough for regularization)

• Convolutional Neural Network (ConvNet)

• Recurrent Neural Network (RNN) + Long-Short Term Memory (LSTM)

- ConvNet has shown outstanding performance in recognition tasks (image, speech)

- ConvNet contains hierarchical abstraction process called pooling.

- RNN+LSTM makes use of long-term memory → Good for time-series data

- RNN is a generative model: It can generate new data

CONTENTS

2. DNN with TensorFlowThanks to Sungjoon Choi

https://github.com/sjchoi86/

DEEP LEARNING LIBRARIES

DEEP LEARNING LIBRARY

• Karpathy’s Recommendation

BASIC WORKFLOW OF TF

1. Load data

2. Define the NN structure

3. Set optimization parameters

4. Run!https://github.com/terryum/TensorFlow_Exercises

EXAMPLE 1

https://github.com/terryum/TensorFlow_Exercises

1. LOAD DATA https://github.com/terryum/TensorFlow_Exercises/blob/master/2_LogisticRegression_MNIST_160516.ipynb

1. LOAD DATA

2. DEFINE THE NN STRUCTURE

3. SET OPTIMIZATION PARAMETERS

4. RUN

4. RUN (C.F.)

EXAMPLE 2

NEURAL NETWORK

Hugo Larochelle, http://www.dmi.usherb.ca/~larocheh/index_en.html

• Activation functions

http://goo.gl/qMQk5H

• Basic NN structure

1. LOAD DATA https://github.com/terryum/TensorFlow_Exercises/blob/master/3a_MLP_MNIST_160516.ipynb

4. RUN

EXAMPLE 3

CONVOLUTION

http://colah.github.io/posts/2014-07-Understanding-Convolutions/

http://www.wildml.com/2015/11/understanding-convolutional-neural-networks-for-nlp/

CONVOLUTIONAL NN

• How can we deal with real images which is much bigger than MNIST digit images?

- Use not fully-connected, but locally-connected NN

- Use convolutions to get various feature maps

- Abstract the results into higher layer by using pooling

- Fine tune with fully-connected NN

https://goo.gl/G7kBjI

https://goo.gl/Xswsbd

http://goo.gl/5OR5oH

1. LOAD DATA https://github.com/terryum/TensorFlow_Exercises/blob/master/4a_CNN_MNIST_160517.ipynb

4. RUN

4. RUN (C.F.)

Thank you

https://www.facebook.com/terryum

http://terryum.io/

http://t-robotics.blogspot.kr/

Introduction to Deep Learning with TensorFlow

Engineering

Machine Learning-Deep Learning Fondement et biaisfiles.meetup.com/19203187/FondetBiaisMLDLonline.pdf · Machine Learning-Deep Learning Fondement et biais Machine learning Aix-Marseille

Deep Learning with FOSS4G

Container + Deep Learning

Einführung ins Deep Learning · Einführung ins Deep Learning deep space computing AG 6. Dezember 2019 deep-space.ch

Introdução ao Deep Learning com o TensorFlow

Introduction to Deep Learning with TensorFlow

Hangzhou Deep Learning Meetup-Deep Reinforcement Learning

Scaling Deep Learning - Nvidiaimages.nvidia.com/.../SC5108-scaling-deep-learning... · Scaling Deep Learning Bryan Catanzaro @ctnzr . Bryan Catanzaro What do we want AI to do?

Solid Day - Deep learning

Docker で Deep Learning

REDES NEURAIS DEEP LEARNING COM TENSORFLOW

AStudyofEvaluationFunctionforPerfectReversi by Deep Learning · its AI is trained by deep learning using existing training data sets. With TensorFlow library that is appeared recently

Zeppelin, TensorFlow, Deep Learning 맛보기

DEEP LEARNING - REVIEW - Pennsylvania State … · Deep Learning - History, Background & Applications. ... Source : Deep learning Yann LeCun, Yoshua Bengio, Geoffrey Hinton Nature

Deep learning 20170311

Deep Learning for Natural Language Processing › slides › 20160618_DL4NLP@CityU.pdf · Machine Learning Deep Learning Implementation 2 Neural Models for Representation Learning

Explanation on Tensorflow example -Deep mnist for expert

Introduction to Deep Reinforcement Learningwnzhang.net/teaching/cs420/slides/13-deep-rl.pdf · 2020-06-08 · Deep Reinforcement Learning •Deep Reinforcement Learning •leverages

Deep Learning基本理論とTensorFlow

Deep Learning : Hopfield Networks