Geecon.cz 2015 debski krzysztof

Krzysztof Dębski, 22nd October 2015

How to handle large amount of events

Geecon.cz 2015


Who am I

15 years as an IT professional

6 years as an Architect in Allegro Group

1 year as a Product Owner http://hermes.allegro.tech

@DebskiChris


Allegro Group

500+ people in IT

50+ independent teams

16 years on market

2 years after technical revolution


Agenda

Choose a tool

Distribute load

Be reliable

Focus on throughput or latency

Improve security

Think ahead


What is important when handling events

Choose a tool

Distribute load

Be reliable


Improve security

Think ahead


Choose a tool


Kafka

Service

Producer

Service

Consumer

KafkaBroker

Zookeeper


Kafka topics

Producer_1…Producer_n

Remove old events

Publish eventTopic


Kafka partitions Producer_1…Producer_n

Publish eventPartition 0

Partition 1

Partition 2


1

Consumer group 2Consumer group 1

Kafka partitions

Consumer 1

Broker 1

P0

Broker 2

P1

Broker 3

P2

Consumer 2 Consumer 3 Consumer 4 Consumer 5


1


Kafka partitions

Consumer 1

Broker 1

P0

Broker 2

P1

Broker 3

P2

Consumer 2 Consumer 3 Consumer 4 Consumer 5 Consumer 6


1


Kafka partitions

Consumer 1

Broker 1

P0

Broker 2

P1

Broker 3

P2

Consumer 2 Consumer 3 Consumer 4 Consumer 5 Consumer 6

Broker 4

P3


Kafka replicas

Service

Producer

Service

Consumer

Broker

Zookeeper

Broker

Broker

P1 P0

P2 P1

P0 P2


Kafka replicas

Service

Producer

Service

Consumer

Broker

Zookeeper

Broker

Broker

P1 P0

P2 P1

P0 P2


HermesHermes Frontend

Hermes Frontend

Hermes Frontend

Hermes Consumer

Hermes ConsumerREST

REST, JMS


Distribute load


Kafka partitions Producer


Partition 1

Partition 2

Default Partitioning - Round Robin




Partition 1

Partition 2

Default Partitioning - Round Robinin practice

Binds to single partition for 10 mins



Key_0Partition 0

Partition 1

Partition 2

Default Partitioning - Key based

Key_1

Key_2


Rebalancing leadersBroker 1

P1 P0

Broker 2

P2 P1

Broker 3

P0 P2

Topic: test Partition count: 3 Replication factor: 1 Configs: retention.ms=86400000

Topic: test Partition: 0 Leader: 3 Replicas: 3, 1 ISR: 3, 1Topic: test Partition: 1 Leader: 1 Replicas: 1, 2 ISR: 1, 2Topic: test Partition: 2 Leader: 2 Replicas: 2, 3 ISR: 2, 3



P1 P0

Broker 2

P2 P1

Broker 3

P0 P2



Brokers that should have partition copies



P1 P0

Broker 2

P2 P1

Broker 3

P0 P2



In Sync Replicas



P1 P0

Broker 2

P2 P1

Broker 3

P0 P2



Leader broker ID



P1 P0

Broker 2

P2 P1

Broker 3

P0 P2





P1 P0

Broker 2

P2 P1

Broker 3

P0 P2


Topic: test Partition: 0 Leader: 1 Replicas: 3, 1 ISR: 1Topic: test Partition: 1 Leader: 1 Replicas: 1, 2 ISR: 1, 2Topic: test Partition: 2 Leader: 2 Replicas: 2, 3 ISR: 2



P1 P0

Broker 2

P2 P1

Broker 3

P0 P2




Be reliable


ACK levels

0 - don’t wait for the response

1 - only the leader has to acknowledge

-1 - all replicas must be in sync

Spee

d

Safe

ty


Lost events

ERROR [Replica Manager on Broker 2]: Error when processing fetch request for partition [test,1] offset 10000 from consumer with correlation id 0. Possible cause:

Request for offset 10000 but we only have log segments in the range 8000 to 9000. (kafka.server.ReplicaManager)


Lost events

Broker 1 Broker 2

Producer

ACK = 1

Replication factor = 1

offset commited = 10000 offset commited = 9000

Zookeeper


Lost events

Broker 1 Broker 2

Producer

ACK = 1

Replication factor = 1

replica.lag.max.messages = 2000


Zookeeper


Lost events

Broker 1 Broker 2


Zookeeper

Producer


Lost events

Broker 1 Broker 2


Zookeeperoffset commited = 9000

Producer


Event identification

Hermes Frontend

KafkaBroker

POST{“event”: ”test”}

{ "id": "58d7ff07-dd0e-4103-9b1f-55706f3049e6", "timestamp”: 1430443071995, “data”: {“event”: ”test”}}

HTTP 201 CreatedMessage-id: 58d7ff07-dd0e-4103-9b1f-55706f3049e6


Lost events

Hermes FrontendProducer

Hermes Consumer

Consumer

KafkaBroker

Zookeeper

Tracker

Publicationdata Delive

ry

attempts


Normal operation


Hermes Consumer

Consumer

KafkaBroker

Zookeeper

POST

HTTP201Created


Abnormal situation


Hermes Consumer

Consumer

KafkaBroker

Zookeeper

POST

HTTP202Accepted




Throughput

Does whole world stop when you stop?


Throughput

Co-ordinated omission


75%

99%

99,9%

resp

onse

tim

e

Slow responses


Slow response vs. message sizem

essa

ge c

size

75%

99%

99,9%


resp

onse

tim

e

75%

99%

99,9%

Slow response and fixed message size


Kafka

kernel 3.2.x


Kafka

kernel 3.2.x


Kafka

kernel 3.2.x kernel >= 3.8.x


Optimize message sizem

essa

ge s

ize

99,9%all topics

99,9%biggest topic


Optimize message size

JSON human readable

big memory and network footprint

poor support for Hadoop



JSON

Snappy

ERROR Error when sending message to topic t3 with key: 4 bytes, value: 100

bytes with error: The server experienced an unexpected error when

processing the request (org.apache.kafka.clients.producer.internals.

ErrorLoggingCallback)

java: target/snappy-1.1.1/snappy.cc:423: char* snappy::internal::

CompressFragment(const char*, size_t, char*, snappy::uint16*, int): Assertion

`0 == memcmp(base, candidate, matched)' failed.

errors on publishing large amount of messages



JSON

Snappy

Lz4

failed on distributed data

com

pres

sion

ratio

single

topic

multiple

topics



JSON

Snappy

Lz4

Avro

small network footprint

Hadoop friendly

easy schema verification


Kafka Offset Monitor


Improve security


Kafka-1682

Kafka <= 0.8.2

No security !

Kafka > 0.8.2

unix-like users, permissions, ACL


Manage your topics

cz.geecon.demo.basicGroup Topic


Improved securityAuthentication and authorization interfaces provided

By Default:

You can create any topic in your group

You can publish everywhere (in progress)

Group owner defines subscriptions


Think ahead


Consumer backoff

You can’t have exactly one delivery

http://bravenewgeek.com/you-cannot-have-exactly-once-delivery/


Improved offset managementHermesProducer

Hermes consumer

Publish event

Commited

Local unsent events

Read event


Improved offset management

Hermes consumer

Local unsent events

New event

Serviceinstance



Hermes consumer

Local unsent events

New event

Serviceinstance

HTTP 503 Unavailable



Local unsent events

New event

Serviceinstance

HTTP 503 Unavailable

Check TTL & Add to queue

Hermes consumer


Consumer backoff

100% adapt 1/s 1/min


Turn back the time

PUT /groups/{group}/topics/{topic}/subscriptions/{subscription}/retransmission -8h


Blog: http://allegro.tech

Twitter: @allegrotechblog

Twitter: @debskichris

Technology

Geecon.cz 2015 debski krzysztof