Eonblast

Erlang and VoltDB

1

Erlang and VoltDB

TechPlanet 2012H. Diedrich

http://www.eonblast.com – twitter @hdiedrich

Eonblast

Erlang and VoltDB

2

Your Host

Henning Diedrich

• Founder, CEO Eonblast• CTO Freshworks• CTO, Producer at Newtracks• Team Lead, Producer at Bigpoint• OS Maintainer Emysql, Erlvolt

Eonblast

Erlang and VoltDB

3

Erlang and VoltDB

1. What makes Them Special?2. Are They For Me? 3. How They Combine4. Getting Started!

Eonblast

Erlang and VoltDB

4

ErlangErlang may be to Javawhat Java was to C++

C++ – pointers = JavaJava – deadlocks = Erlang

Eonblast

Erlang and VoltDB

5

Erlang was Built For

• Reliability• Maintenance• Distribution• Productivity

Eonblast

Erlang and VoltDB

6

Erlang Poster Childs

Klarna AB• Financial Services for E-Commerce• 30 seconds downtime in 3 years

Distributed Databases• Membase• Riak• BigCouch

Eonblast

Erlang and VoltDB

7

Sweet Spots

• Stateful Servers with High Throughput• Cluster Distribution Layers

Eonblast

Erlang and VoltDB

8

The Magic

• Microprocesses• Pattern Matching*

• Immutable Variables

* Not your familiar Regex string matching

Eonblast

Erlang and VoltDB

9

The Actor ModelCarl Hewitt 1973• Behavior• State• Parallel• Asynchronous Messages• Mailboxes• No Shared State• Self-Contained Machines

Benefits• More true to the real world• Better suited for parallel hardware• Better suited for distributed architectures• Scaling garbage collection (sic!)• Less Magic

Data Code

Object

Data Code

Actor

Process

Eonblast

Erlang and VoltDB

10

Thinking ProcessesWhat should be a Process?

Processes• Don’t share State• Communicate Asynchronously• Are Very Cheap to create And keep• Monitor Each Other• Provide Contention Handling• Constitute the Error Handling Atom

„It's easy!“ Joe Armstrong

Eonblast

Erlang and VoltDB

11

Objects and Threads

Objects sharing Threads, Object Lifetime, Idle Threads

Eonblast

Erlang and VoltDB

12

Erlang Actors

Erlang Actors: State + Code + Process

Eonblast

Erlang and VoltDB

13

Processes are Transactional

One actor is one process and so, cannot “race itself”.Mandating a job kind to an actor creates a transactional funnel.Only one such job will ever be executing at any one time.

Do X1 for me!

Do X2 for me!

Doing X1 Doing X2Funnel

Eonblast

Erlang and VoltDB

14

Thinking Parallel

„It's not easy.“

Robert Virding

• The Generals’ Problem• Lamport Clocks• No Guarantees

Eonblast

Erlang and VoltDB

15

Thinking Functional

Small Functions

+ Immutable Variables

→ Don’t assign variables: return results!

Complete State in Plain Sight

→ Awful for updates in place.

→ Awsome for debugging & maintenance.

Erlang is not side-effect free at all.

Eonblast

Erlang and VoltDB

16

Let It Crash!

• No Defense Code• On Error, restart Entire Process• Built-In Process Supervision & Restart• Missing Branches, Matches cause Crash

→ Shorter, Cleaner Code

→ Faster Implementation

→ More Robust: handles All Errors

Eonblast

Erlang and VoltDB

17

Syntax

* Small

* Easy

* Stable

* Declarative

* Inspired by Prolog and ML

* Obvious State, Implicit Thread

fib(0) -> 0;

fib(1) -> 1;

fib(N) when N>1 -> fib(N-1) + fib(N-2).

Eonblast

Erlang and VoltDB

18

Hello, World! -module(hello).-export([start/0, loop/0]).

start() -> Pid = spawn(hello, loop, []), Pid ! hello.

loop() -> receive hello -> io:format("Hello, World!~n"), loop() end.

From Edward Garson's Blog at http://egarson.blogspot.de/2008/03/real-erlang-hello-world.html

Eonblast

Erlang and VoltDB

19

Immutable VariablesCan’t assign a second time:

A = A + 1.

A = 1, A = 2.

* Prevent Coding Errors

* Provide Transactional Semantic

* Allow for Pattern Matching Syntax

* Can be a Nuisance

Eonblast

Erlang and VoltDB

20

Pattern Matching

This can mean two things:

A = func().

The meaning depends on whetherA is already assigned.

Eonblast

Erlang and VoltDB

21

Pattern MatchingThe common, mixed case:

{ok, A} = func().

ok is an assertion AND

A is being assigned.

Eonblast

Erlang and VoltDB

22

Proven Productivity

Motorola Study of 2002 – 2006:

“Erlang shows …• 2x higher throughput• 3x better latency• 3 - 7x shorter code… than the equivalent C++ implementation.”

Eonblast

Erlang and VoltDB

23

VoltDB

The free, scaling, SQL DB

MySQL + scale = VoltDB

Eonblast

Erlang and VoltDB

24

CAP

• Distributed• Consistent• Highly-Available• Partition-Tolerant

… really? all of it!Brewer on CAP 2012: http://www.infoq.com/articles/cap-twelve-years-later-how-the-rules-have-changed

Eonblast

Erlang and VoltDB

25

ACID

• Atomicity• Consistency• Isolation• Durability

… for granted?

Eonblast

Erlang and VoltDB

26

Double Bookkeeping

• Not Every App needs It• Requires ACID Transactions• Neigh Impossible to emulate• Impossible With BASE

(Eventual Consistency)

Eonblast

Erlang and VoltDB

27

• VoltDB, Inc. 2009 – commercial developer, support• Open Source – 100% dictatorial• Made for OLTP – fast cheap writes, high throughput• CA of CAP – 100% consistent & highly available• Simple SQL – subset of SQL '92• ACID transactions – double bookkeeping• In-memory – 100x faster than MySQL• Distributed – painless growth• Linear scale – predictable, low cost• Replication, Snapshots – disk persistence, hot backup• More SQL than SQL – clean separation of data

VoltDB

Eonblast

Erlang and VoltDB

28

In-Memory

• Today good for 100s of GB of data• The Redis of clusters• Sheds 75% of DBM activity• Full disk persistence

Eonblast

Erlang and VoltDB

29

Snowflake Structures

Data

Data

Data

Data

Eonblast

Erlang and VoltDB

30

Partitions

State#

a-za-z

a-za-z a-z

A B CE F G H I J K L

M

N O P

Eonblast

Erlang and VoltDB

31

Erlvolt• Erlang VoltDB Driver• Open Source• Asynchronous

InsertConnection = erlvolt:createConnection("localhost", "program", "password"),

erlvolt:callProcedure(Connection, "Insert", ["안녕하세요", "세계", "Korean"]),

SelectResponse = erlvolt:callProcedure(Connection, "Select", ["Korean"]),

Row = erlvolt:fetchRow(Table, 1),

io:format("~n~n~s, ~s!~n",

[ erlvolt:getString(Row, Table, "HELLO"),

erlvolt:getString(Row, Table, "WORLD") ]);

Eonblast

Erlang and VoltDB

32

Benchmark

• Amazon EC2• 64 core node.js clusters + 96 core VoltDB cluster• 695,000 transactions per second (TPS)• 2,780,000 operations per second• 100,000 TPS per 8 core client• 12,500 TPS per node.js core• Stable even under overload• Pretty much linear scale

Eonblast

Erlang and VoltDB

33

Benchmark

// Check if the vote is for a valid contestant

SELECT contestant_number FROM contestants WHERE contestant_number = ?;

// Check if the voter has exceeded their allowed number of votes

SELECT num_votes FROM v_votes_by_phone_number WHERE phone_number = ?;

// Check an area code to retrieve the corresponding state

SELECT state FROM area_code_state WHERE area_code = ?;

// Record a vote

INSERT INTO votes (phone_number, state, contestant_number) VALUES (?, ?, ?);

Eonblast

Erlang and VoltDB

34

ResourcesErlang

Webhttp://www.erlang.org/ http://learnyousomeerlang.com/http://www.erlang.org/static/getting_started_quickly.html

Listhttp://erlang.org/mailman/listinfo/erlang-questionshttp://groups.google.com/group/erlang-programming

Bookshttp://pragprog.com/book/jaerlang/programming-erlanghttp://shop.oreilly.com/product/9780596518189.do

Referenceshttp://erldocs.com/ http://www.erlang.org/doc/

Post Mortemshttp://www.facebook.com/note.php?note_id=14218138919 http://www.slideshare.net/wooga/erlang-the-big-switch-in-social-games

Erlvolt

https://github.com/Eonblast/Erlvolt

VoltDB

Downloadhttp://voltdb.com/products-services/downloads

Forumhttp://community.voltdb.com/forum

Voter Examplehttps://github.com/VoltDB/voltdb/tree/master/examples/voter

Benchmark Blog Posthttp://voltdb.com/company/blog/695k-tps-nodejs-and-voltdb

Volt’s Magic Saucehttp://nms.csail.mit.edu/~stavros/pubs/hstore.pdf

Eonblast

Erlang and VoltDB

35

Questions

• Email: hdiedrich*eonblast.com• Twitter: @hdiedrich• IRC: #erlounge• List: erlang-questions@erlang.org