Come diventare data scientist - Paolo Pellegrini

BIG DATA & DATA SCIENCE

COME DIVENTARE UN DATA SCIENTISTPRIMI CONSIGLI PER STUDENTI E PERSONE ALLE PRIME ARMI

Paolo Pellegrini, Senior Consultant

giugno 2016


Perché diventare un Data Scientist?

Chi è un Data Scientist?

Come viene selezionato un Data Scientist?

Cos’è un algoritmo?

Un esempio di problema di Data Science

Alcuni Tool di Data Science

Dove aggiornarsi e formarsi?

1

AGENDA


IL 56% DELLE IMPRESE ITALIANE INDICA BIG DATA E DATA SCIENCE COME PRIMARIO SVILUPPO STRATEGICO PER IL 2016/17

2

3%5%6%

7%10%10%

17%17%18%18%

25%25%

31%31%

40%48%

53%56%

0% 10% 20% 30% 40% 50% 60%

Smart ManufacturingInternet of Things

Smart WorkingProgetti commerciali web social

Cyber SecurityCompliance e Risk Management

CollaborationStorage e virtualizzazione

Mobile e eCommerceData Center

Mobile Marketing e CRMCloud pubblico e privato

Consolidamento applicativoSistemi CRM

Device Mobili e Mobile AppsSistemi ERP

DematerializzazioneBig Data e Analytics


HARVARD, GIA’ ANNI FA, LO AVEVA DEFINITO IL LAVORO PIÙ SEXY DEL NOSTRO SECOLO…ED È ANCHE BEN REMUNERATO!

3

GOOGLE TREND «DATA SCIENTIST»

AVERAGE SALARY 123,000 $









4

AGENDA


UN DATA SCIENTIST PUÒ FARE TUTTO!

5

Nate Silver è la persona che ha cambiato ilconcetto di “Psephology”, usando Big Data& Data Science per predire i risultatidelle elezioni Americane.Oggi, è uno dei più famosi Data Scientistal mondo


UN DATA SCIENTIST È UNA FIGURA FORTMENTEINTERDISCIPLINARE, CHE CONIUGA STATISTICA, PROGRAMMAZIONE E LOGICHE DI BUSINESS

6

«On any given day a team member might author a multistage processing pipeline in Python, design a

hypothesis test, perform a regression analysis over data samples with R, design and implement an

algorithm for some data-intensive product or service in Hadoop, or communicate the results of an

analysis to other members of the organization in a clear and concise fashion»

2009 – Jeff Hammerbacher | Data Scientist @ Facebook


LA DIFFUSIONE NEL MONDO È SEMPRE PIÙ GRANDE, E VEDE L’ITALIA PROTAGONISTA

7

RJ Metrics on 11.400 Data Scientist profile on LinkedIn


I SETTORI CHE IMPIEGANO PIÙ DATA SCIENTISTSONO QUELLI MAGGIORMENTE ORIENTATI ALL’IT, MA LA DIFFUSIONE E’ SEMPRE PIÙ CAPILLARE

8



LE COMPETENZE PIÙ DIFFUSE VERTONO SU LINGUAGGI E STRUMENTI COME «R» E «PYTHON»

9



LE COMPETENZE MODELLISITICHE E DI PROGRAMMAZIONE SONO FONDAMENTALI PER UNA RISORSA JUNIOR

10



UN DATA SCIENTIST PUÒ AVERE QUALSIASI TIPO DI BACKGROUND: CONTA SOLO VOGLIA E ATTITUDINE A LAVORARE SUI DATI

11



2%11%

14%

73%

PRESTO OGNI AZIENDA AVRÀ UN DATA SCIENTIST

12

Present, with a well defined rolePresent, but without a well defined roleIntroduction planned for 2016Possible introduction in the future

Data Scientist









13

AGENDA


PERCORSO DI VALUTAZIONE

14

Job posting

Test development

CV selection

Test assignment

Test evaluation

Technical interview

Data Scientist evaluation



15

Job posting

Test development

CV selection

Test assignment

Test evaluation

Technical interview


1) You have two tables in an existing RDBMS. One containsinformation about the products you sell (name, size, color, etc.)The other contains images of the products in JPEG format.These tables are frequently joined in queries to your database.You would like to move this data into HBase. What is the mostefficient schema design for this scenario?

• Create a single table, with two column family• Create a single table, with one column family• Create two tables, with one column family

2) A sandwich shop studies the number of men, and women, thatenter the shop during the lunch hour from noon to 1pm each day.They find that the number of men that enter can be modeled as arandom variable with distribution Poisson(M), and likewise thenumber of women that enter as Poisson(W). What is likely to bethe best model of the total number of customers that enter duringthe lunch hour?

• Poisson (M+W)• Possion (M/W)• Poisson (M*W)

Junion Data Scientist Selection



16

Job posting

Test development

CV selection

Test assignment

Test evaluation

Technical interview


Senior Data Scientist Selection

Consegna di un Data Set, via mail o tramite piattaformecome University2Business, che i candidate devonoanalizzare al fine di sviluppare un modello predittivo



17

Job posting

Test development

CV selection

Test assignment

Test evaluation

Technical interview


Seguono domande generiche sulla costruzione dei modelli odiscussioni di dettaglio su quando svolto nel test. Ad esempio:• Pulizia dati• Costruzione modello• Sviluppo algoritmo• …

Senior Data Scientist Selection



18

Job posting

Test development

CV selection

Test assignment

Test evaluation

Technical interview


Socio-economic

Statistic

Business

MathSoft – Rolespecific

Soft – Businessgeneric

Computer Science


DESCRIZIONE DELLE COMPETENZE

19

TEC

HN

ICA

L SK

ILLSS

OFT

RO

LE S

PE

CIFIC

SO

FT B

US

INE

SS

GE

NE

RIC

Socio-Economiche

Capacità di lettura del contesto sociale e di come questo impatti

sul contesto economico

SettorialiConoscenza di

processi, mercato e anticipazione degli

impatti delle variabili esogene sullo specifico

settore

MatematicheCapacità di sistemizzare

la realtà attraverso classificazioni e modelli che tengano conto delle

interazioni fra gli elementi

InformaticheCapacità di trattamento

dell’informazione, mediante lo sviluppo di procedure

automatizzate (es. algoritmi) e di un supporto

HW/SW

StatisticheCapacità trarre

deduzioni logiche ed estrarre conoscenza

dallo studio di un particolare fenomeno

non deterministico

HackingCapacità di fare uso di

creatività e immaginazione nella ricerca della

conoscenza

StorytellingCapacità di inventiva nella creazione di scenari da

esplorare e di inserire le informazioni all’interno di un framework che ne facilitino la trasmissione e la

comprensione all’esterno, anche attraverso capacità di sintesi e di presentazione delle informazioni

EticaCapacità di fare uso

coscienzioso dei dati, anche a fronte del possesso di dati

sensibili

ManagementCapacità di guida e

coordinamento di un gruppo di risorse, assunzione di decisioni

per garantire l'ottenimento di risultati aziendali

TeamworkCapacità di operare in gruppo, attraverso spartizione di ruoli e

aggregazione di competenze, al fine di raggiungere un obiettivo

comune

Coaching/MentoringCapacità di formazione di

risorse con meno esperienza, al fine di migliorarne le

potenzialità, partendo dall’unicità dell'individuo

Relazioni interpersonaliCapacità di relazionarsi con altri

soggetti, ponendosi nel modo opportuno a seconda di status,

relazioni gerarchiche, contingenze, ecc.


PROFILI TIPICI

20

Junior Data Scientist

Socio-Economic

Statistic

Business

MathSoft Role Specic

Soft Bsuiness Generic

Computer Science

Socio-Economic

Statistic

Business



Computer Science

Senior Data Scientist

Chief Data Scientist

Socio-Economic

Statistic

Business



Computer Science

Business Manager

Socio-Economic

Statistic

Business



Computer Science









21

AGENDA


“Data is inherently dumb - Algorithms are where the real value lies. Algorithms define action”

Peter SondergaardSenior Vice President

Gartner Research

DAL DATO ALL’ALGORITMO

22

A

graphical expression of Euclid's algorithm to find the greatest common divisor for 1599 and 650

Algorithm is a self-contained step-by-step set of operations to be performed


COME GLI ALGORITMI SUPPORTANO IL BUSINESS

23

INFORMATION INSIGHTS DECISION ACTION

DESCRIPTIVE

Whathappened?

DIAGNOSTIC

Why does ithappened?

PREDICTIVE

What future?

PRESCRIPTIVE

How to react to recent events?

PREEMPTIVE

How to avoidbad events?

DATA-DRIVENSTRATEGY

DecisionalSupport

OPTIMIZATION STRATEGY

ANALYTICSSTRATEGY

OLD-STYLE STRATEGY

+


• Machine learning is a subfield of computer science, that evolved from the study of pattern recognition and computational learning theory in artificial intelligence

• In 1959, Arthur Samuel defined machine learning as a "Field of study that gives computers the ability to learn without being explicitly programmed“

• Machine learning explores the study and construction of algorithms that can learn from and make predictions on data. Such algorithms operate by building a model from an example training set of input observations in order to make data-driven predictions or decisions expressed as outputs rather than following strictly static program instructions

• Machine learning is closely related to (and often overlaps with) computational statistics; a discipline which also focuses in prediction-making through the use of computers. It has strong ties to mathematical optimization, which delivers methods, theory and application domains to the field

COME SI DIFINISCE IL «MACHINE LEARNING»

24


• C4.5 - Constructs a classifier in the form of a decision tree. In order to do this, C4.5 is given a set of data representing things that are already classified. This is supervised learning, since the training dataset is labeled with classes

ALCUNI ESEMPI DI ALGORITMI

25


• k-means - creates k groups from a set of objects so that the members of a group are more similar. It’s a popular cluster analysis technique for exploring a dataset. Most would classify k-means as unsupervised. Other than specifying the number of clusters, k-means “learns” the clusters on its own without any information about which cluster an observation belongs to k-means can be semi-supervised


26


• Support vector machines - SVM teaches a hyperplane to classify data into 2 classes. At a high-level, SVM performs a similar task like C4.5 except SVM doesn’t use decision trees at all. It is a supervised learning, since a dataset is used to first teach the SVM about the classes


27


• Naive Bayes - it is not a single algorithm, but a family of classification algorithms that share one common assumption: every feature of the data being classified is independent of all other features given the class. This is supervised learning, since Naive Bayes is provided a labeled training dataset in order to construct the tables


28


• PCA - Principal component analysis uses an orthogonal transformation to convert a set of observations of possibly correlated variables into a set of values of linearly uncorrelated variables called principal components. The number of principal components is less than or equal to the number of original variables. The first principal component has the largest possible variance, and each succeeding component in turn has the highest variance possible under the constraint that it is orthogonal to the preceding components. The resulting vectors are an uncorrelated orthogonal basis set. The principal components are orthogonal because they are the eigenvectors of the covariance matrix, which is symmetric. PCA is sensitive to the relative scaling of the original variables. This is unsupervised learning


29


PROCESSO LOGICO DI USO DEGLI ALGORITMI

30

Ricezione Dataset

Analisi esplorativa dei dati Pulizia dei dati

Uso di Algoritmi per trovare variabili più

predittive

Costruzione modello logico

• Random Forest• Decision Tree• SVM• …

Testing & TunionSviluppo algoritmo ad-hoc di supporto

al business









31

AGENDA


DATA SCIENTIST: MAGICIAN OR SUPERHERO?

32

Può un Data Scientist predire i crimini a San Francisco?Può un Data Scientist aiutare la città ad esser più sicura?


From 1934 to 1963, San Francisco was infamous for housing some of the world's most notorious criminals on the inescapable island of Alcatraz.Today, the city is known more for its tech scene than its criminal past. But, with rising wealth inequality, housing shortages, and a proliferation of expensive digital toys riding BART to work, there is no scarcity of crime in the city by the bay.

• It’s provided a dataset of 12 years of incidents from across all of San Francisco's neighborhoods, from 1/1/2003 to 13/05/2015.

• Dataset has been divided in two parts: a training set, to be used for the model development, and a test set, useful to verify the predictive algorithm.

THE SAN FRANCISCO CHALLENGE

33

Given time and location, you must predict the category of crime that can occur


878.049 INCIDENTS WITH

39 CATEGORIES OF CRIME

TRAINING SET STRUCTURE

34

For every incident is provided:

• Data and time

• Category

• Description

• Day of week

• Pd District

• Resolution

• Address

• Latitude

• Longitude


UN ESEMPIO DI DATA VISUALIZATION

35


Q1 – HOW TO ACT WITH THE DATASET?

36

• To manage Dataset are used CSV files, but also JSON. No xls! • 800.000 record is Big Data! • You can use only variables known when the model is applied

Variable: Data and time, Day of week, Pd District, Address, Latitude, Longitude

Variable not to be included: Description, Resolution

Taget: Category


Manage Dataset in order to make all valid variable usable by a predictive model:

• Generate an ID for every record• Verify the structure of every variable and search for data that need to be

cleaned up (e.g. empty record, double space, ecc.)• Split “Date” (13/05/2015 23:53:00) into single variables (Month, Year, Hour)• Merge “Latitude” and “Longitude” to verify the presence of unique place

ÆVerify the distribution of every variable to find out the presence of “non-normal distribution” or other kind of problems to be fix

STEP 1: DATA CLEANSING

37


STEP 2: LAUNCH THE FIRST EXPLORATIVE MODEL

38

You can use professional (and free) tools like Rapid Miner, Weka, Knime, ecc.


• Tool: IBM Watson

• Algorithm: Decision Tree CHAID

• Predictive Strength: 17% - less than 1/4 category crime is correctly predicted

STEP 2: LAUNCH THE FIRST EXPLORATIVE MODEL

39

Top PredictorsDecision Tree for Arson


Q1 – HOW TO ACT WITH THE DATASET?

40

• DataÆWeekend&HolidayDummy [using historical calendar holidays]• TimeÆ NightDummy, Cold/RainDummy, HotDummy,

WorkingTimeDummy [using weather, sunborn/set, ecc. information]• Address Æ StreetType [managing strings]• OtherÆ UnemploymentRateByMonth, VisitorsRateByMonth,

PoupulationDensityByDistrict, HouseCostByDistrict, EducationLevelByDistrict [desk analysis]

• Imagine how to transform Data variable• Imagine how to transform Time variable• Imagine how to transform Address variable• Imagine other external information to be included in the model• Try to select sources to import these information


STEP 3: LAUNCH THE FINAL EXPLORATIVE MODEL

41

Top Predictors

• Tool: IBM Watson

• Algorithm: Decision Tree CHAID

• Predictive Strength: 32% - about 1/3 category crime is correctly predicted









42

AGENDA


BIG DATA = TANTISSIME TECNOLOGIE

43


SCEGLIERE UN TOOL DI DATA SCIENCE

44

Linguaggi• R• Phyton

Altri Tool• HP Vertica• Weka• Tableau• Neo4K

COMPLETENESS OF VISION Æ

ABI

LITY

TO

EXE

CUTE

Æ

CHALLENGERS

NICHE PLAYER

LEADERS

VISIONARIES

SAS

IBM KNIME

RapidMiner

MicrosoftAlteryx

Predixion Software

Alphine Data

FICO

LavastormMegaputer

Prognoz

Accenture

DellSAPAngoss

MAGIC QUADRANT 2016 - GARTNER

Pay

Free

LEGEN

DA


RAPID MINER: UN LABORATORIO VISUALE

45


IBM WATSON: UNO DEGLI STRUMENTI PIÙ FAMOSI A MENO DI 50€/MESE PER UTENTE

46









47

AGENDA


MOOC

48


COME DIVENTARE UN DATA SCIENTISTPRIMI CONSIGLI PER STUDENTI E PERSONE ALLE PRIME ARMI

Paolo Pellegrini, Senior Consultant

giugno 2016