Deep Learning

Mohammad Ali Keyvanrad

Lecture 1:Introduction

OUTLINE • Recent success with Deep Learning

• Deep Learning definition

• History

• Course plan

• Resources

• Grading Policy

9/24/2017 M.A Keyvanrad Deep Learning (Lecture1-Introduction) 2

OUTLINE • Recent success with Deep Learning

• Deep Learning definition

• History

• Course plan

• Resources

• Grading Policy

9/24/2017 M.A Keyvanrad Deep Learning (Lecture1-Introduction) 3

9/24/2017 M.A Keyvanrad Deep Learning (Lecture1-Introduction) 4

Recent success with Deep Learning

• Over the last few years Deep Learning was applied tohundreds of problems. Computer vision and pattern recognition

Speech recognition and speech synthesis

Natural language processing

Computer games, robots & self-driving cars

…

• In many problems they have established the state ofthe art Often exceeding previous benchmarks by large margins

9/24/2017 M.A Keyvanrad Deep Learning (Lecture1-Introduction) 5

Learning Lip Sync from Audio(University of Washington, 2017)

9/24/2017 M.A Keyvanrad Deep Learning (Lecture1-Introduction) 6

Restore colors in B&W photos (Waseda University, 2016)

Input [Larsson et al. 2016] [Zhang et al. 2016a] [Zhang et al. 2016b]

9/24/2017 M.A Keyvanrad Deep Learning (Lecture1-Introduction) 7

Pixel restoration(Google Brain, 2017)

• Take very low resolution images and predict what eachimage most likely looks like.

9/24/2017 M.A Keyvanrad Deep Learning (Lecture1-Introduction) 8

Describing photos(Stanford University, 2015)

• Computers can automatically classify our photos Facebook can automatically tag your friends

• Deep Learning not only learned to classify the elementsin the photo, but to actually describe them.

9/24/2017 M.A Keyvanrad Deep Learning (Lecture1-Introduction) 9

Translation(Google Translate, 2015)

• Google Translate app now does real-time visualtranslation of 20 more languages. A photo taken by the phone, and Google Translate "reads"

the text and replaces it with a text in English in real-time.

9/24/2017 M.A Keyvanrad Deep Learning (Lecture1-Introduction) 10

Create new images(University of California, 2017)

• Deep Learning network to create other types of newimages

9/24/2017 M.A Keyvanrad Deep Learning (Lecture1-Introduction) 11

Reading text in the Wild(University of Oxford, 2014)

• An attempt to read text from photos and videos Search for text from BBC News videos

9/24/2017 M.A Keyvanrad Deep Learning (Lecture1-Introduction) 12

Teach a computer to play(DeepMind, 2015)

• Google's DeepMind used a Deep Learning technique toteach a computer to play Control of the keyboard while watching the score, and its

goal was to maximize the score

9/24/2017 M.A Keyvanrad Deep Learning (Lecture1-Introduction) 13

Beating people in dozens of computer games

Computer program playing Doom using only raw pixel data.

9/24/2017 M.A Keyvanrad Deep Learning (Lecture1-Introduction) 14

Self-driving cars(Tesla, 2016)

• A Tesla electric vehicle drives without humanintervention Notice how it distinguishes different type of objects,

including people and road signs.

9/24/2017 M.A Keyvanrad Deep Learning (Lecture1-Introduction) 15

Robotics(BostonDynamics, 2016)

• Deep Learning is also heavily used in robotics thesedays SpotMini and Atlas

9/24/2017 M.A Keyvanrad Deep Learning (Lecture1-Introduction) 16

Voice generation(Google WaveNet, 2016)

• Deep Learning is taking us a step closer to givingcomputers the ability to speak like humans do. Google released WaveNet and Tacotron

Baidu released Deep Speech

9/24/2017 M.A Keyvanrad Deep Learning (Lecture1-Introduction)

• Tacotron learns pronunciations based on phrasesemantics. “He has read the whole thing.”

“He reads books.”

• Tacotron is sensitive to punctuation. “This is your personal assistant, Google Home.”

• Tacotron learns stress and intonation. “The buses aren't the problem, they actually provide a solution.”

“The buses aren't the PROBLEM, they actually provide a SOLUTION.”

• Tacotron's prosody changes in a question. “The quick brown fox jumps over the lazy dog.”

“Does the quick brown fox jump over the lazy dog?”

17

Voice generation(Google Tacotron, 2017)

9/24/2017 M.A Keyvanrad Deep Learning (Lecture1-Introduction) 18

Restoring sound in videos(MIT, Berkeley, Google, 2016)

• Deep Learning network was trained on videos in whichpeople were hitting and scratching objects

• After several iterations learning, the scientists asked thecomputer to regenerate the sound

9/24/2017 M.A Keyvanrad Deep Learning (Lecture1-Introduction) 19

LIPNET(Oxford, DeepMind, 2016)

• LipNet reached 93% success in reading people's lipswhere an average lipreader succeeds 52% of the time.

9/24/2017 M.A Keyvanrad Deep Learning (Lecture1-Introduction) 20

Automatically writing(Stanford University, 2016)

• Let a Deep Learning network "read" Shakespeare,Wikipedia, math papers and computer code.

9/24/2017 M.A Keyvanrad Deep Learning (Lecture1-Introduction) 21

Handwriting(University of Toronto, 2014)

• Today the computer can also handwrite.

OUTLINE • Recent success with Deep Learning

• Deep Learning definition

• History

• Course plan

• Resources

• Grading Policy

9/24/2017 M.A Keyvanrad Deep Learning (Lecture1-Introduction) 22

9/24/2017 M.A Keyvanrad Deep Learning (Lecture1-Introduction) 23

Deep Learning definition

• Deep learning is a class of machine learning algorithmsthat: Definition 1: They use a cascade of many layers of nonlinear processing units for

feature extraction and transformation Each successive layer uses the output from the previous layer as

input. The algorithms may be supervised or unsupervised. Applications include pattern analysis (unsupervised) and

classification (supervised).

Definition 2: They are part of the broader machine learning field of learning

representations of data.

9/24/2017 M.A Keyvanrad Deep Learning (Lecture1-Introduction) 24

Deep Learning definition

Definition 3: These are based on the (unsupervised) learning of multiple levels

of features or representations of the data.

Higher level features are derived from lower level features to forma hierarchical representation.

Definition 4: They learn multiple levels of representations that correspond to

different levels of abstraction; the levels form a hierarchy ofconcepts.

9/24/2017 M.A Keyvanrad Deep Learning (Lecture1-Introduction) 25

Deep Learning definition

• Common in definitions Models consisting of multiple layers or stages of nonlinear

information processing.

The supervised or unsupervised learning of featurerepresentations in each layer, with the layers forming ahierarchy from low-level to high-level features.

9/24/2017 M.A Keyvanrad Deep Learning (Lecture1-Introduction) 26

Deep Learning definition

• Deep or Shallow? Credit Assignment Path (CAP) A chain of transformations from input to output.

CAPs describe potentially causal connections between input andoutput.

CAP depth Number of hidden layers plus one

as the output layer is also parameterized

For recurrent neural networks the CAP depth is potentiallyunlimited.

a signal may propagate through a layer more than once.

9/24/2017 M.A Keyvanrad Deep Learning (Lecture1-Introduction) 27

Deep Learning definition

• Deep/shallow?

• No universally agreed upon threshold of depth divides shallowlearning from deep learning

• Most researchers agree that deep learning has multiple nonlinearlayers (CAP > 2).

OUTLINE • Recent success with Deep Learning

• Deep Learning definition

• History

• Course plan

• Resources

• Grading Policy

9/24/2017 M.A Keyvanrad Deep Learning (Lecture1-Introduction) 28

9/24/2017 M.A Keyvanrad Deep Learning (Lecture1-Introduction) 29

History

9/24/2017 M.A Keyvanrad Deep Learning (Lecture1-Introduction) 30

Evolution of DepthLeNet

1998

7 Layers

AlexNet

2012

8 Layers

GoogLeNet

2014

22 Layers

ResNet

2016

152 Layers

9/24/2017 M.A Keyvanrad Deep Learning (Lecture1-Introduction) 31

Evolution of Depth

OUTLINE • Recent success with Deep Learning

• Deep Learning definition

• History

• Course plan

• Resources

• Grading Policy

9/24/2017 M.A Keyvanrad Deep Learning (Lecture1-Introduction) 32

9/24/2017 M.A Keyvanrad Deep Learning (Lecture1-Introduction) 33

Course plan

• Introduction

• A Review of Artificial Neural Networks Perceptron

Stochastic Gradient Descent

Backpropagation

Rectified Linear Function

Root Mean Square Propagation

Dropout

L1 and L2 Regularization

9/24/2017 M.A Keyvanrad Deep Learning (Lecture1-Introduction) 34

Course plan

• Deep Belief Network PGM

MRF

Sampling

RBM

• Auto-Encoder Linear Auto-Encoder

Denoising Auto-Encoder

• Computational Network

9/24/2017 M.A Keyvanrad Deep Learning (Lecture1-Introduction) 35

Course plan

• Selected Applications in Object Recognition and ComputerVision Convolutional Neural Networks Region Based CNN Generative Adversarial Network GoogLeNet and Microsoft ResNet

• Selected Applications in Language Modeling and NaturalLanguage Processing Word2Vec Recurrent Neural Networks and Language Models Machine translation and advanced recurrent LSTMs and GRUs

9/24/2017 M.A Keyvanrad Deep Learning (Lecture1-Introduction) 36

Course plan

• Selected Applications in Speech and Audio Processing Speech recognition and bi-directional RNN

Speech synthesis and WaveNet or Tacotron

• Deep Reinforcement Learning

OUTLINE • Recent success with Deep Learning

• Deep Learning definition

• History

• Course plan

• Resources

• Grading Policy

9/24/2017 M.A Keyvanrad Deep Learning (Lecture1-Introduction) 37

9/24/2017 M.A Keyvanrad Deep Learning (Lecture1-Introduction) 38

Resources

• Ian Goodfellow, Yoshua Bengio, Aaron Courville, “Deep Learning (AdaptiveComputation and Machine Learning series)”, MIT Press, 2016.

• Dong Yu, Li Deng, “Automatic Speech Recognition: A Deep LearningApproach”, Springer, 2015

• L. Deng and D. Yu, “Deep Learning: Methods and Applications”, NowPublishers Inc, 2014.

• Pattern Recognition and Machine Learning, Christopher M. Bishop, 2006.

• Stanford (CS224n: Natural Language Processing with Deep Learning, 2017)

• Stanford (CS231n: Convolutional Neural Networks for Visual Recognition,2017)

• Related papers

OUTLINE • Recent success with Deep Learning

• Deep Learning definition

• History

• Course plan

• Resources

• Grading Policy

9/24/2017 M.A Keyvanrad Deep Learning (Lecture1-Introduction) 39

9/24/2017 M.A Keyvanrad Deep Learning (Lecture1-Introduction) 40

Grading Policy

• Assignments: 20%

• Presentation: 15%

• Final Exam: 35%

• Final Project: 30%

9/24/2017 M.A Keyvanrad Deep Learning (Lecture1-Introduction) 41

9/24/2017 M.A Keyvanrad Deep Learning (Lecture1-Introduction) 42