H2O PySparkling Water

H2O PySparkling Water

Michal Malohlava @mmalohlava and @h2oai

presents

2016/10/08 PyData

H2O.aiMachine Intelligence

H2O+

PySpark =

PySparkling


H2OOpen-Source In-Memory Data Science Platform

•Highly optimized Java code (in-house)

•Distributed in-memory K-V store and map/reduce computation framework

•Data parser (HDFS, S3, NFS, HTTP, local drives, etc.)

•Read/write access to distributed data frames (R/Pandas-style)

•ML algos - Deep Learning, GBM, DRF, GLM, GLRM, K-Means, PCA, CoxPH, Ensembles

•REST API: clients Interactive UI/R/Python

H2O Python client

pySparkling


PySparklingProvides

Transparent integration of H2O machine learning platform with Spark ecosystem (PySpark)

Transparent use of H2O data structures (H2OFrame) and algorithms with Spark Python API

Excels in existing Spark workflows requiring advanced Machine Learning algorithms

Func

tiona

lity

mis

sing

in H

2O c

an b

e re

plac

ed b

y Sp

ark

and

vice

ver

sa


Benefits

• Additional algorithms

• NLP features

• Powerful data munging

• ML Pipelines

• Advanced algorithms

• speed v. accuracy

• advanced parameters

• Fully distributed and parallelized

• Graphical environment

• Fully fledged Python/R interfaces


PySparklingUse-Cases


Model Building

Data Source


Model Building

Data Source

Data munging


Model Building

Data Source

Data munging Modelling

Deep Learning, GBMDRF, GLM, GLRM

K-Means, PCACoxPH, Ensembles


Model Building

Data Source




Prediction processing


Model Building

Data Source




Prediction processing

Steam

Model management


Data Munging

Data Source


Data Munging

Data Source


Data Munging

Data Source


Data Munging

Data Source

Data load/munging/ exploration (H2O Flow UI)


Data Munging

Data Source

Data load/munging/ exploration (H2O Flow UI) Modelling


Stream processing


Stream processing

DataSourceO

ff-lin

e m

odel

trai

ning


Stream processing

DataSourceO

ff-lin

e m

odel

trai

ning

Data munging


Stream processing

DataSourceO

ff-lin

e m

odel

trai

ning



Stream processing

DataSourceO

ff-lin

e m

odel

trai

ning

Data mungingSt

ream

proc

essi

ng

Data Stream

Spark Streaming/Storm/Flink

Modelling


Stream processing

DataSourceO

ff-lin

e m

odel

trai

ning

Data mungingSt

ream

proc

essi

ng

Data Stream

Spark Streaming/Storm/Flink

Export model

Modelling


Stream processing

DataSourceO

ff-lin

e m

odel

trai

ning

Data mungingSt

ream

proc

essi

ng

Data Stream

Spark Streaming/Storm/FlinkModel

prediction

Deploy the model

Export model

Modelling


Stream processing

DataSourceO

ff-lin

e m

odel

trai

ning

Data mungingSt

ream

proc

essi

ng

Data Stream

Spark Streaming/Storm/FlinkModel

prediction

Deploy the model

Export model

Modelling

Stea

m

Mod

el

man

agem

ent


What is inside?


Cluster

Worker node

PySpark main program

Driver node Worker nodeWorker node


Cluster

Worker node


Driver node

SparkContext

Worker nodeWorker node

sc = SparkContext.getOrCreate()


Cluster

Worker node

Spark executor Spark executorSpark executor


Driver node

SparkContext




Cluster

Worker node



Driver node

SparkContext




Cluster

Worker node



Driver node

SparkContext


H2OContext


h2o_context = H2OContext.getOrCreate()


Cluster

Worker node



Driver node

SparkContext


H2OContext


h2o_context = H2OContext.getOrCreate()


DataSource

Spar

k Ex

ecut

orSp

ark

Exec

utor

Spar

k Ex

ecut

or

Spark Cluster

DataFrame

H2O

Sto

re H2OFrame

DataSource

H2O

Sto

reH

2O S

toreDat

a In

gest


DataSource

Spar

k Ex

ecut

orSp

ark

Exec

utor

Spar

k Ex

ecut

or

Spark Cluster

DataFrame

H2O

Sto

re H2OFrame

h2o_context.as_spark_frame H2OFrame serves data for DataFrame operations

DataSource

H2O

Sto

reH

2O S

tore

Dat

a Ex

chan

ge

PyAPI


DataSource

Spar

k Ex

ecut

orSp

ark

Exec

utor

Spar

k Ex

ecut

or

Spark Cluster

DataFrame

H2O

Sto

re H2OFrame

h2o_context.as_h2o_frame Materializes DataFrame as H2OFrame (H2O as a clever cache)

DataSource

H2O

Sto

reH

2O S

tore

Dat

a Ex

chan

ge

PyAPI


Sentiment Analysis

with PySparklingDEM

O


Start PySparkling

Opens Jupyter Notebook

Download from h2o.ai/download


Future


The PlanSeparation of H2O cluster from Spark infrastructure ✓ Preserving existing API

h2oContext = H2OContext.getOrCreate(ip=“…”, port=…)

Better integration into PySpark pipelines ✓ Support of H2O Ensembles (right now only as R-package)

Integration with Steam platform to support model management DeepWater integration H2O DeepWater with Python

early sneakFabrizio MiloSu

nday

3p

m


Checkout GitHub & Contribute

https://github.com/h2oai/sparkling-water

Checkout H2O.ai Training Books http://h2o.ai/resources

Checkout H2O.ai Blog http://h2o.ai/blog/

Checkout H2O.ai Youtube Channel https://www.youtube.com/user/0xdata

More info

https://www.youtube.com/user/0xdata


Learn more at h2o.ai Follow us at @h2oai

Come to see us at Open Tour in Dallas! See open.h2o.ai

PySparkling is open-source

ML application platform combining

power of PySpark and H2O

Thank you!

DALLAS, TX OCT 26th

Engineering

H2O PySparkling Water