Apache Spark on EMR

Yuyang Lan

SmartNews Inc.

MOKUJI

• Intro

• Recent Spark

• How we use Spark in Smartnews

• Best Practices

Who am I

• @y2_lan

• Engineer at SmartNews Inc. (AD team)

• Hacker, Data Engineer, Beer Lover

何か要望・問題あったら @kaiseh :)

About Apache Spark

maybe

just skip?

About Apache Spark Quick catch up

RDD

action

transformations

Recent Spark at a glance

• Databricks Cloud goes public

• Spark 1.4.x

• Project Tungsten

• AWS adds support for Apache Spark on EMR

• …

Spark 1.4.x• SparkR

• DataFrame API

• ML Pipeline

• Streaming UI

• …

Spark at SmartNews

• AD CTR Prediction ( Logistic Regression )

Spark at SmartNews

• Scoring articles by Kinesis + Spark Streaming

Spark at SmartNews

• Ad-Hoc Analysis, Faster (& Hive-compatible) SQL

Spark at SmartNews

• Realtime Stats by Kinesis + Spark Streaming

Spark at SmartNews• ML experiments

• AD targeting

• User Clustering

• Recommendation

• …

Best Practices #1

• Should use the default Spark with EMR ?

• Yes Sure

• EMR 4.0 is great ! (Released today ?!)

• Hadoop 2.6 + Hive 1.0 + Spark 1.4.1

Best Practices #1

• Should use the default Spark with EMR ?

• But only if you need a custom-build Spark

• Cutting Edge Version

• Native netlib-java ( mvn -Pnetlib-lgpl )

• Custom dependency version

• …

Best Practices #1

• Should use the default Spark with EMR ?

• But only if you need a custom-build Spark

• --bootstrap-actions bootstrap.json

Best Practices #1

• Should use the default Spark with EMR ?

• But only if you need a custom-build Spark

• Remember to start SparkHistoryServer

Best Practices #2

• Run Spark on Yarn

• Use yarn-cluster mode to distribute Drivers

• specify jars and files to distribute necessary resources

Best Practices #3

• Tuning Memory

• CPU shortage only slow down your program, but short in memory make it crash

• you can even set --executor-cores bigger than your CPU num

• Cache-able heap != JVM’s Xmx

• (normally about 50%)

Best Practices #3

• Tuning Memory

• CPU shortage only slow down your program, but short in memory make it crash

• Cache-able heap != JVM’s Xmx

Image from: http://0x0fff.com/spark-architecture/

Best Practices #3

• Tuning Memory

• CPU shortage only slow down your program, but short in memory make it crash

• Cache-able heap != JVM’s Xmx

• spark.yarn.executor.memoryOverhead

• spark.executor.memory

• spark.storage.memoryFraction

• …

• Split your executors if HEAP_SIZE > 64GB (GC)

• -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCTimeStamps

Best Practices #4

• If your ML job is really CPU-bound

• Try using OpenBLAS + netlib.NativeSystemBLAS

Best Practices #4

• Try using OpenBLAS + netlib.NativeSystemBLAS

4~5 times FAST

Best Practices #5

• Minimize data shuffle

• Prefer reduceByKey over groupByKey+map

• RDD.repartition(NUM_OF_CORES) before cache

• Try to do filter early

Best Practices #5

• Minimize data shuffle

Best Practices #6

• Prefer DataFrame APIs over low level RDD APIs

• Better DAG Optimization

• Same interface & same performance

Best Practices #7

• Use Kryo serialization if possible

--conf spark.serializer=org.apache.spark.serializer.KryoSerializer

Best Practices #8

• Pick up a notebook tool (iPython or Zeppelin or ?

• For memo, sharing, visualisation

• Convenient for non-engineer users

Best Practices #9

• Multiple small & task-driven EMR clusters

Best Practices #10

• use Dynamic scaling with Spark Streaming

• spark.dynamicAllocation.enabled = true

• spark.shuffle.service.enabled = true

• be careful if you use cached data

Best Practices #11

• Use Spot Instance

• Be more aggressive in bid price : p

• BID_PRICE != MONEY_TO_PAY

• Check Spot Instance Pricing History

• Find the instance type with relative stable price

• often Previous Generation Instance ?

• Prepare failure, don’t use them in critical missions

Further Reading

• To use Spark Streaming in Production

• http://www.slideshare.net/SparkSummit/recipes-for-running-spark-streaming-apploications-in-production-tathagata-daspptx

Further Reading

• If you’re interested in new ML pipelines

• http://www.slideshare.net/SparkSummit/building-debugging-and-tuning-spark-machine-leaning-pipelinesjoseph-bradley

Thanks!

We’re hiring!

http://about.smartnews.com/ja/careers/

iOSエンジニア / Androidエンジニア / Webアプリケーションエンジニア

/ プロダクティビティエンジニア / 機械学習 / 自然言語処理エンジニア

/ グロースハックエンジニア / サーバサイドエンジニア

/ 広告エンジニア…