Machine Learning, Deep Learning and Data Analysis Introduction

PowerPoint

Machine Learning, Deep Learning and Data Analysis

ITRI CONFIDENTIAL DOCUMENT DO NOT COPY OR DISTRIBUTE

ITRI CONFIDENTIAL DOCUMENT DO NOT COPY OR DISTRIBUTEOutline2Overview of ML, DL and Data Analysis What is Machine LearningTake a Look At Linear RegressionOther ML Algorithms at a GlanceWhat is Neural Network?What is Deep Learning?Deep Learning using TensorFlowData AnalysisCase 1, 2 and 3Multivariate Analysis


2

My BackgroundFAB/Mobile Phone/Device

System/Application DevelopmentEDA

Validation ToolData Center

Cloud IaaS / PaaS developmentMaster of Computer and Information Science (Cleveland State University, USA)Master of Industrial Engineering (NCTU, Taiwan) 0.5 year1 year3 yearsNetworking

Switch L2/L3 ProtocolsAnd SDN2 years6 yearsITRI

ML/DLBig Data Analysis


My Experience for Machine Learning4!!Hope giving you an experience and guidelineTake courses:Coursera: Machine Learning ( Got Certificate )Udemy: Data Science: Deep Learning in Python ( ongoing)Study on-line resources:YoutubeML/DL tutorials and so onhttps://morvanzhou.github.io/http://bangqu.com/gpu/bloghttp://www.jiqizhixin.com/insightsGet you hands dirtyPython programmingTensorFlow Deep Learning LibraryScikit-Learn LibraryNumby, Pandas, matplotlib,


From AI to Deep Learning5: 2017 Google https://www.youtube.com/watch?v=uZ-7DVzRCy8

https://blogs.nvidia.com.tw/2016/07/whats-difference-artificial-intelligence-machine-learning-deep-learning-ai/CPU/GPUBig DataAlgorithmsBreakthrough

ITRI CONFIDENTIAL DOCUMENT DO NOT COPY OR DISTRIBUTEML, DL and Data Analysis6

Visually LinkingWhat we focus todayhttps://whatsthebigdata.com/2016/10/17/visually-linking-ai-machine-learning-deep-learning-big-data-and-data-science/

??Data Analysis

ITRI CONFIDENTIAL DOCUMENT DO NOT COPY OR DISTRIBUTEWhat is Machine Learning

7

ITRI CONFIDENTIAL DOCUMENT DO NOT COPY OR DISTRIBUTEMachine Learning definition8Arthur Samuel (1959). Machine Learning: Field of study that gives computers the ability to learn without being explicitly programmed.Tom Mitchell (1998) Well-posed Learning Problem: A computer program is said to learn from experience E with respect to some task T and some performance measure P, if its performance on T, as measured by P, improves with experience E.

ITRI CONFIDENTIAL DOCUMENT DO NOT COPY OR DISTRIBUTEMachine Learning definition9Suppose your email program watches which emails you do or do not mark as spam, and based on that learns how to better filter spam. What is the task T in this setting? Classifying emails as spam or not spam. (T)Watching you label emails as spam or not spam. (E)The number (or fraction) of emails correctly classified as spam/not spam. (P)None of the abovethis is not a machine learning problem

ITRI CONFIDENTIAL DOCUMENT DO NOT COPY OR DISTRIBUTEWhat is Machine Learning ?Without writing any custom code specific to the problemFeed data to the generic algorithmIt builds its own logic

ITRI CONFIDENTIAL DOCUMENT DO NOT COPY OR DISTRIBUTETwo styles of Machine LearningSupervised Learning

Unsupervised Learning

Use the logic to predict the sales price

figure out if there is a pattern or grouping or something

FeaturesLabel

ITRI CONFIDENTIAL DOCUMENT DO NOT COPY OR DISTRIBUTEWhat are machine learning algorithms?Regression AlgorithmsLinear RegressionLogistic RegressionLASSODecision Tree AlgorithmsClassification and Regression Tree (CART)Iterative Dichotomiser 3 (ID3)C4.5 and C5.0 (different versions of a powerful approach)Bayesian AlgorithmsNaive BayesClustering Algorithms (unsupervised)k-MeansSupport Vector MachinesPrincipal Component AnalysisAnomaly DetectionRecommender SystemsArtificial Neural Network Algorithms

ITRI CONFIDENTIAL DOCUMENT DO NOT COPY OR DISTRIBUTELets Take a Look At Linear Regression13

ITRI CONFIDENTIAL DOCUMENT DO NOT COPY OR DISTRIBUTELinear Regression14The Hypothesis Function

Cost Function

Gradient Descent for Multiple Variables

https://www.coursera.org/learn/machine-learning/

ITRI CONFIDENTIAL DOCUMENT DO NOT COPY OR DISTRIBUTEGradient Descent15How to choose learning


ITRI CONFIDENTIAL DOCUMENT DO NOT COPY OR DISTRIBUTEGradient Descent16Convergence of gradient descent with an appropriate learning rate

Cost Function https://www.coursera.org/learn/machine-learning/

ITRI CONFIDENTIAL DOCUMENT DO NOT COPY OR DISTRIBUTELinear Regression17Training data with linear regression fit


ITRI CONFIDENTIAL DOCUMENT DO NOT COPY OR DISTRIBUTEOther ML Algorithms at a Glance18

ITRI CONFIDENTIAL DOCUMENT DO NOT COPY OR DISTRIBUTELogistic Regression19Training data with decision boundary

linear decision boundaryno linear decision boundaryhttps://www.coursera.org/learn/machine-learning/

ITRI CONFIDENTIAL DOCUMENT DO NOT COPY OR DISTRIBUTESupport Vector Machines20The difference between the kernels in SVMLinearPolynomialGaussian (RBF)SigmoidSVM (Gaussian Kernel) Decision BoundaryChoose gamma ( auto )

Gaussian (RBF)Non-linear decision boundaryhttps://www.coursera.org/learn/machine-learning/

ITRI CONFIDENTIAL DOCUMENT DO NOT COPY OR DISTRIBUTEK-Means21The original 128x128 image with 24-bit color (three 8-bit )using K-means (K=16) to use find the 16 colors that best group (cluster) the pixels in the 3-dimensional RGB space.

K=3 and computing centroid means Iterativelyhttps://www.coursera.org/learn/machine-learning/

ITRI CONFIDENTIAL DOCUMENT DO NOT COPY OR DISTRIBUTEPrincipal Component AnalysisAn example to deal with image dimension reduction and proximate recovery.

Faces Dataset

Recovered faces

Principal components

ITRI CONFIDENTIAL DOCUMENT DO NOT COPY OR DISTRIBUTEWhat is Neural Network(We will review the previous concepts a little bit)23

ITRI CONFIDENTIAL DOCUMENT DO NOT COPY OR DISTRIBUTEML -- write that program by ourselvesTo estimate the price of a houseIf we could just figure out the perfect weights to use that work for every house, our function could predict house prices!How to do that with ML?def estimate_house_sales_price(num_of_bedrooms, sqft, neighborhood):

price = 0 # a little pinch of this

price += num_of_bedrooms * .841231951398213

price += sqft * 1231.1231231

price += neighborhood * 2.3242341421

price += 201.23432095

return price

ITRI CONFIDENTIAL DOCUMENT DO NOT COPY OR DISTRIBUTEML -- write that program by ourselvesStep 1 Initialize weights to 1.0Step 2See the difference and how far off the function is at guessing the correct price

Step 3Repeat Step 2 over and over with every single possible combination of weights.

ITRI CONFIDENTIAL DOCUMENT DO NOT COPY OR DISTRIBUTEML -- What about that whole try every number bit in Step 3?

is what represents your current weights. J() means the cost for your current weights.Clever waysto quickly find good values for those weights without having to try very many.If we graph this cost equation for all possible values of our weights for number_of_bedroomsandsqftGradient Descent


26

Making SmarterGuesses27We ended up with this simple estimation function

def estimate_house_sales_price(num_of_bedrooms, sqft, neighborhood):

price = 0 # a little pinch of this

price += num_of_bedrooms * 0.123

price += sqft * 0.41

price += neighborhood * 0.57

return price

alinearrelationship with the input

ITRI CONFIDENTIAL DOCUMENT DO NOT COPY OR DISTRIBUTEIf there is more complicated situation?Different of weights for the different house sizes

ITRI CONFIDENTIAL DOCUMENT DO NOT COPY OR DISTRIBUTEWhat is a Neural NetworkNow we have four different price estimates. Lets combine those four price estimates into one final estimate.

neuronsThis is a neural network

ITRI CONFIDENTIAL DOCUMENT DO NOT COPY OR DISTRIBUTEWhat is a Neural NetworkHuman Brains

http://www.slideshare.net/tw_dsconf/ss-62245351

ITRI CONFIDENTIAL DOCUMENT DO NOT COPY OR DISTRIBUTEWhat is a Neural Network

Different connections lead to different structured network.http://www.slideshare.net/tw_dsconf/ss-62245351


31

Fully Connected Feedforward Network


ITRI CONFIDENTIAL DOCUMENT DO NOT COPY OR DISTRIBUTEFully Connected Feedforward Network

Deep Learninghttp://www.slideshare.net/tw_dsconf/ss-62245351

ITRI CONFIDENTIAL DOCUMENT DO NOT COPY OR DISTRIBUTEFully connected feedforward networkMatrix Operation


ITRI CONFIDENTIAL DOCUMENT DO NOT COPY OR DISTRIBUTEOutput LayerSoftmax layer as the output layer

ITRI CONFIDENTIAL DOCUMENT DO NOT COPY OR DISTRIBUTENeural Network Playgroundhttp://playground.tensorflow.org/

ITRI CONFIDENTIAL DOCUMENT DO NOT COPY OR DISTRIBUTEWhat is Deep Learning?

37

ITRI CONFIDENTIAL DOCUMENT DO NOT COPY OR DISTRIBUTEWhat is Deep Learning?38Deep Learning is Large Neural NetworksDeep Learning attracts lots of attention

http://static.googleusercontent.com/media/research.google.com/en//people/jeff/BayLearn2015.pdf

ITRI CONFIDENTIAL DOCUMENT DO NOT COPY OR DISTRIBUTEWhy Deep Learning?The more data, the more performance.

Game ChangerDL accuracy/performance is more than 99%


39

Deep Learning Models40Convolutional Neural NetworkInception-V3Recurrent Neural NetworkLSTMAuto-encoderReinforcement LearningQ-LearningPolicy GradientWide and Deep LearningRecommender system

ITRI CONFIDENTIAL DOCUMENT DO NOT COPY OR DISTRIBUTEDeep Learning is not so simpleBackpropagation an efficient way to compute Gradient DescentOverfittingChoosing Loss functionSquare Error, Cross Entropy, and so onMini-Batch Too deep ( many hidden layers )ReLU, MaxOut, Learning RatesMomentumAdam ( optimizer )Weight DecayDropout

ITRI CONFIDENTIAL DOCUMENT DO NOT COPY OR DISTRIBUTEBackpropagation42A common method of trainingartificial neural networksand used in conjunction with anoptimization methodsuch asgradient descent.

ITRI CONFIDENTIAL DOCUMENT DO NOT COPY OR DISTRIBUTEUnderfitting and OverfittingBias-Variance Tradeoff

ITRI CONFIDENTIAL DOCUMENT DO NOT COPY OR DISTRIBUTEConvolutional Neural Network (CNN)Why CNN is for image?The first layer of fully connected network would be very large

ITRI CONFIDENTIAL DOCUMENT DO NOT COPY OR DISTRIBUTEThe solution is Convolution

ITRI CONFIDENTIAL DOCUMENT DO NOT COPY OR DISTRIBUTEThe solution is Convolution

ITRI CONFIDENTIAL DOCUMENT DO NOT COPY OR DISTRIBUTEConvolutional Neural Network (CNN)

Adding Even MoreSteps


47

Convolutional Neural Network (CNN)

ITRI CONFIDENTIAL DOCUMENT DO NOT COPY OR DISTRIBUTEDeep Learning using TensorFlow

49

ITRI CONFIDENTIAL DOCUMENT DO NOT COPY OR DISTRIBUTELinear Regression in TensorFlow50

X_data array([ 0. , 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1. , 1.1, 1.2, 1.3, 1.4, 1.5, 1.6, 1.7, 1.8, 1.9, 2. , 2.1, 2.2, 2.3, 2.4, 2.5, 2.6, Y_data array([ 0. , 0.29999667, 0.59997333, 0.89991 , 1.19978668, 1.49958339, 1.79928013, 2.09885695, 2.39829388, 2.69757098, 2.99666833, 3.29556602,



ITRI CONFIDENTIAL DOCUMENT DO NOT COPY OR DISTRIBUTEMNIST53MNIST dataset55000 samples (50000 for training, 5000 for testing)For each sample, it has X, y parts. X are the image with 28*28 pixels in 8 bit gray scaleY is the label answer: 0, 1, 2, , 9

ITRI CONFIDENTIAL DOCUMENT DO NOT COPY OR DISTRIBUTEMNIST54X, y can be represented as follows

ITRI CONFIDENTIAL DOCUMENT DO NOT COPY OR DISTRIBUTEMNIST55

If you want to get the accuracy more than 99%, check it out:https://gotocon.com/dl/goto-london-2016/slides/MartinGorner_TensorflowAndDeepLearningWithoutAPhD.pdf92%

ITRI CONFIDENTIAL DOCUMENT DO NOT COPY OR DISTRIBUTEImage Recognition and Retraining Inception-v3 model is ready and made by Googleit took Google researchers two weeks to build on a desktop with eight NVidia Tesla K40s.It can recognize > 1000 categoriesRetrainingTo prepare the new images and categoriesDo training and testing

ITRI CONFIDENTIAL DOCUMENT DO NOT COPY OR DISTRIBUTEPlate Number RecognitionThere is an example using UKs Plate Number and Character to train TensorFlow CNN modelTake 3 days with GPU Card (GTX 750 TI)

http://matthewearl.github.io/2016/05/06/cnn-anpr/

ITRI CONFIDENTIAL DOCUMENT DO NOT COPY OR DISTRIBUTETechnical breakthrough for Deep-ANPR

http://matthewearl.github.io/2016/05/06/cnn-anpr/

ITRI CONFIDENTIAL DOCUMENT DO NOT COPY OR DISTRIBUTEAutoencoderEnccode the input data (MNIST data) and then decode it backIt is similar to PCA

autoencoderoriginal

ITRI CONFIDENTIAL DOCUMENT DO NOT COPY OR DISTRIBUTELSTM (RNN)It is a special kind of RNN, capable of learning long-term dependencies

LSTM Training with

ITRI CONFIDENTIAL DOCUMENT DO NOT COPY OR DISTRIBUTERecurrent Neural Network61

RNN Modeltraining

outputPlay this map with Super Mario Makerhttps://medium.com/@ageitgey/machine-learning-is-fun-part-2-a26a10b68df3


61

GPU62Setup NVIDIA Quadro M6000 GPU CardInstall driver (NVIDIA-Linux-x86_64-375.26.run)CUDA Tookit 8.0 (cuda-repo-ubuntu1404-8-0-local_8.0.44-1_amd64.deb)CuDNN v5.1 (cudnn-8.0-linux-x64-v5.1.tgz)The M6000s performance could be affected by not enough power supplyOnly 1.3 ~ 1.7 Times faster than GTX 750 Ti

ITRI CONFIDENTIAL DOCUMENT DO NOT COPY OR DISTRIBUTE63

Quadro M6000GTX 750 Ti


ITRI CONFIDENTIAL DOCUMENT DO NOT COPY OR DISTRIBUTEAOI Defect Classification64

ITRI CONFIDENTIAL DOCUMENT DO NOT COPY OR DISTRIBUTEA deepCNN model for AOI65Image recognition model, Inception-v3To achieve reasonable performance on hard visual recognition tasks

Defect Classification

ITRI CONFIDENTIAL DOCUMENT DO NOT COPY OR DISTRIBUTE(Big) Data Analysis

66

ITRI CONFIDENTIAL DOCUMENT DO NOT COPY OR DISTRIBUTEData Analysis67The steps to do data analysisData CollectionFrom CSV files, database, and so on.Data Preprocessing ( very very very important )Regularization, Normalization, Table Join, Feature ExtractionReduce the dimensions .Feature SelectionSelect the important featuresMachine Learning / Deep LearningTo train the modelTo do the Prediction and Classification by the trained modelApply or implement to systemBut, still needs:domain experts involved!!Studying related papers and researches

ITRI CONFIDENTIAL DOCUMENT DO NOT COPY OR DISTRIBUTEAnalysis Tools and Libraries68Open Sources(Python)Machine LearningSciKit-LearnNumPyMatplotlibPandasDeep LearningTensorFlowKerasHadoop & SparkCommercial Software ( rare to use)PolyAnalyst 6.5SAS

ITRI CONFIDENTIAL DOCUMENT DO NOT COPY OR DISTRIBUTEData Analysis69In my experience with data analysis, I belong to a Rookie


ITRI CONFIDENTIAL DOCUMENT DO NOT COPY OR DISTRIBUTEData Analysis Case 1

70

ITRI CONFIDENTIAL DOCUMENT DO NOT COPY OR DISTRIBUTEESL in Details71Trainingpython nn_esl1.py --do_training True --epoch 350Prediction:python nn_esl1.py --x1 10 --x2 25 --x3 0.2 --x4 20 --epoch 350('Result: ', array([[ 145.28657532, 193.63188171]], dtype=float32))Original Label Result Data: [145.69, 190.95]5%Feature SelectionThreshold = 0.25 array([0, 1]) D, PThreshold = 0.1 array([0, 1, 3])D, P, H

> 500 raw data



ITRI CONFIDENTIAL DOCUMENT DO NOT COPY OR DISTRIBUTE73ESL Web App Architecture 73Dell Server(Ubuntu 14.04) VirtualBoxVirtual Machine (Ubuntu 14.04)NVIDIA Quadro M6000 GPU Card (12GB)

ESL Web App

UsersTensorFlowTrained Model

TensorFlow

2Apply the trained ESL Model

3Query

Criteria: Scenario, Timing, Power, Electrical-SI/Pi, Thermal, Mechanical

1Train ESL Model Output:

2.5D TSVPoP2.5D Fan-OutInFO PoP

4Predict the module design


ITRI CONFIDENTIAL DOCUMENT DO NOT COPY OR DISTRIBUTEData Analysis Case 2

74

ITRI CONFIDENTIAL DOCUMENT DO NOT COPY OR DISTRIBUTE75Bond Time, Power, and Force Pull

#9(y = x * w + b) w = np.array([[ 0.31186092], [ 0.26080179], [ 0.19781925]])b = np.array([[ 0.22946352]])

# Normalization $ teX = np.array([[150/np.max(data[:,0]), 95/np.max(data[:,1]), 500/np.max(data[:,2])]]) array([[ 0.83333333, 1.0, 0.71428571]])# Prediction $ teX.dot(w) + barray([[ 0.89144887]])# $ (teX.dot(w) + b) * 952array([[ 848.65932832]]) : 842



9Pull


ITRI CONFIDENTIAL DOCUMENT DO NOT COPY OR DISTRIBUTEMultivariate Analysis

77

ITRI CONFIDENTIAL DOCUMENT DO NOT COPY OR DISTRIBUTEMultivariate Analysis78


ITRI CONFIDENTIAL DOCUMENT DO NOT COPY OR DISTRIBUTEReferenceMachine Learning is Fun! MediumMachine Learning is Fun! Part 2 MediumMachine Learning is Fun! Part 3: Deep Learning and ... - MediumDeep Learning TutorialFIRST CONTACT WITH TENSORFLOWhttps://ireneli.eu/2016/02/03/deep-learning-05-talk-about-convolutional-neural-network%EF%BC%88cnn%EF%BC%89/http://www.slideshare.net/tw_dsconf/ss-71780267https://morvanzhou.github.io/tutorials/python-basic/https://media.readthedocs.org/pdf/python-for-multivariate-analysis/latest/python-for-multivariate-analysis.pdfhttp://blog.topspeedsnail.com/http://www.leiphone.com/news/201702/vJpJqREn7EyoAd09.htmlPython scikit-learnhttps://machine-learning-python.kspax.io/version >= 0.17

ITRI CONFIDENTIAL DOCUMENT DO NOT COPY OR DISTRIBUTEThank You80

,


Data & Analytics

Machine Learning, Deep Learning and Data Analysis Introduction