Continuous Integration for Spark Apps by Sean McIntyre

Continuous Integrationfor Spark Apps

Hi, I’m Sean!

© 2015 Uncharted Software Inc.

It’s hard to test Spark Apps :(


Case Study: Uncharted Spark Pipeline


Case Study: Uncharted Spark PipelineSome key issues:

● Ensure reliability● Prevent regressions● Maintain compatibility with multiple versions of Spark● Open-source - need a quick and easy way to evaluate PRs


What is Continuous Integration?


“Continuous Integration (CI) is a development practice that requires developers to integrate code into a shared repository several times a day. Each check-in is then verified by an

automated build, allowing teams to detect problems early.”

-- ThoughtWorks


“Continuous Integration (CI) is a development practice that is pretty damnedimportant for writing quality software.”

-- Me


So, What is Continuous Integration?


Best Practices, courtesy of Wikipedia1. Maintain a code repository (Git)


Best Practices, courtesy of Wikipedia1. Maintain a code repository (Git)2. Automate the build (Gradle)


Best Practices, courtesy of Wikipedia1. Maintain a code repository (Git)2. Automate the build (Gradle)3. Tests should be part of the build (ScalaTest)


Best Practices, courtesy of Wikipedia1. Maintain a code repository (Git)2. Automate the build (Gradle)3. Tests should be part of the build (ScalaTest)4. Commit/push feature branches often


Best Practices, courtesy of Wikipedia1. Maintain a code repository (Git)2. Automate the build (Gradle)3. Tests should be part of the build (ScalaTest)4. Commit/push feature branches often5. Build (and test) All The Branches


Best Practices, courtesy of Wikipedia1. Maintain a code repository (Git)2. Automate the build (Gradle)3. Tests should be part of the build (ScalaTest)4. Commit/push feature branches often5. Build (and test) All The Branches6. Test in a clone of the production environment


Best Practices, courtesy of Wikipedia1. Maintain a code repository (Git)2. Automate the build (Gradle)3. Tests should be part of the build (ScalaTest)4. Commit/push feature branches often5. Build (and test) All The Branches6. Test in a clone of the production environment7. Keep the build fast


Best Practices, courtesy of Wikipedia1. Maintain a code repository (Git)2. Automate the build (Gradle)3. Tests should be part of the build (ScalaTest)4. Commit/push feature branches often5. Build (and test) All The Branches6. Test in a clone of the production environment7. Keep the build fast8. Everyone can see the results of builds



} duh.



} ...less duh.


Why are these difficult with Apache Spark?

5. Build (and test) All The Branches6. Test in a clone of the production

environment7. Keep the build fast8. Everyone can see the results of builds


What is a Spark App?


What is a Spark app?

Source JARSpark ?

This thing.

JAR


And...

Source JARSpark ?

We need to test this

JAR


But...

Source JARScalaTestScala RE

By default, we have this

JAR

(boom)


v1: Squish Spark inside ScalaTest

Source JAR

ScalaTest with

SparkContext

So, we try this

JAR

it works!(sort of)


it works!(sort of)


6. Test in a clone of the production environment


v2: Squish ScalaTest into Spark

Source

TestJAR

Tests Main.scala

Spark

JAR TestJAR

Test Output

JAR


Main.scala


6. Test in a clone of the production environment


Progress?

5. Build (and test) All The Branches6. Test in a clone of the production environment7. Keep the build fast8. Everyone can see the results of builds


What now?



Docker Container (uncharted/sparklet)

v3: Squish Spark and Test JAR into Docker

Test Output

Source

TestJAR

Tests Main.scala

Spark

JAR

JAR TestJAR


test.sh


build.gradle (excerpt)


Progress?



Travis CI VM

Docker Container

v4: Squish Docker into Travis CI

Test Output

Source

TestJAR

Tests Main.scala

Spark

JAR

JAR TestJAR


.travis.yml


Voilà!


Progress?





All done!



Next Steps?

Alpine Linux

docker-compose

Windows (dev environment) support

python


Questions?

https://github.com/unchartedsoftware/sparkpipe-core

https://github.com/Ghnuberath

@Ghnuberath

https://hub.docker.com/r/uncharted/sparklet/

[email protected]

Data & Analytics

Continuous Integration for Spark Apps by Sean McIntyre