Geecon.cz 2015 debski krzysztof

Krzysztof Dębski, 22nd October 2015

How to handle large amount of events

Geecon.cz 2015

Who am I

15 years as an IT professional

6 years as an Architect in Allegro Group

1 year as a Product Owner http://hermes.allegro.tech

@DebskiChris

Allegro Group

500+ people in IT

50+ independent teams

16 years on market

2 years after technical revolution

Agenda

Choose a tool

Distribute load

Be reliable

Focus on throughput or latency

Improve security

Think ahead

What is important when handling events

Choose a tool

Distribute load

Be reliable

Improve security

Think ahead

Choose a tool

Service

Producer

Service

Consumer

KafkaBroker

Zookeeper

Kafka topics

Producer_1…Producer_n

Remove old events

Publish eventTopic

Kafka partitions Producer_1…Producer_n

Publish eventPartition 0

Partition 1

Partition 2

Consumer group 2Consumer group 1

Kafka partitions

Consumer 1

Broker 1

Broker 2

Broker 3

Consumer 2 Consumer 3 Consumer 4 Consumer 5

Kafka partitions

Consumer 1

Broker 1

Broker 2

Broker 3

Consumer 2 Consumer 3 Consumer 4 Consumer 5 Consumer 6

Kafka partitions

Consumer 1

Broker 1

Broker 2

Broker 3

Consumer 2 Consumer 3 Consumer 4 Consumer 5 Consumer 6

Broker 4

Kafka replicas

Service

Producer

Service

Consumer

Broker

Zookeeper

Broker

Kafka replicas

Service

Producer

Service

Consumer

Broker

Zookeeper

Broker

HermesHermes Frontend

Hermes Frontend

Hermes Consumer

Hermes ConsumerREST

REST, JMS

Distribute load

Kafka partitions Producer

Partition 1

Partition 2

Default Partitioning - Round Robin

Partition 1

Partition 2

Default Partitioning - Round Robinin practice

Binds to single partition for 10 mins

Key_0Partition 0

Partition 1

Partition 2

Default Partitioning - Key based

Rebalancing leadersBroker 1

Broker 2

Broker 3

Topic: test Partition count: 3 Replication factor: 1 Configs: retention.ms=86400000

Topic: test Partition: 0 Leader: 3 Replicas: 3, 1 ISR: 3, 1Topic: test Partition: 1 Leader: 1 Replicas: 1, 2 ISR: 1, 2Topic: test Partition: 2 Leader: 2 Replicas: 2, 3 ISR: 2, 3

Broker 2

Broker 3

Brokers that should have partition copies

Broker 2

Broker 3

In Sync Replicas

Broker 2

Broker 3

Leader broker ID

Broker 2

Broker 3

Broker 2

Broker 3

Topic: test Partition: 0 Leader: 1 Replicas: 3, 1 ISR: 1Topic: test Partition: 1 Leader: 1 Replicas: 1, 2 ISR: 1, 2Topic: test Partition: 2 Leader: 2 Replicas: 2, 3 ISR: 2

Broker 2

Broker 3

Be reliable

ACK levels

0 - don’t wait for the response

1 - only the leader has to acknowledge

-1 - all replicas must be in sync

Lost events

ERROR [Replica Manager on Broker 2]: Error when processing fetch request for partition [test,1] offset 10000 from consumer with correlation id 0. Possible cause:

Request for offset 10000 but we only have log segments in the range 8000 to 9000. (kafka.server.ReplicaManager)

Lost events

Broker 1 Broker 2

Producer

ACK = 1

Replication factor = 1

offset commited = 10000 offset commited = 9000

Zookeeper

Lost events

Broker 1 Broker 2

Producer

ACK = 1

Replication factor = 1

replica.lag.max.messages = 2000

Zookeeper

Lost events

Broker 1 Broker 2

Zookeeper

Producer

Lost events

Broker 1 Broker 2

Zookeeperoffset commited = 9000

Producer

Event identification

Hermes Frontend

KafkaBroker

POST{“event”: ”test”}

{ "id": "58d7ff07-dd0e-4103-9b1f-55706f3049e6", "timestamp”: 1430443071995, “data”: {“event”: ”test”}}

HTTP 201 CreatedMessage-id: 58d7ff07-dd0e-4103-9b1f-55706f3049e6

Lost events

Hermes FrontendProducer

Hermes Consumer

Consumer

KafkaBroker

Zookeeper

Tracker

Publicationdata Delive

attempts

Normal operation

Hermes Consumer

Consumer

KafkaBroker

Zookeeper

HTTP201Created

Abnormal situation

Hermes Consumer

Consumer

KafkaBroker

Zookeeper

HTTP202Accepted

Throughput

Does whole world stop when you stop?

Throughput

Co-ordinated omission

Slow responses

Slow response vs. message sizem

Slow response and fixed message size

kernel 3.2.x

kernel 3.2.x kernel >= 3.8.x

Optimize message sizem

99,9%all topics

99,9%biggest topic

Optimize message size

JSON human readable

big memory and network footprint

poor support for Hadoop

Snappy

ERROR Error when sending message to topic t3 with key: 4 bytes, value: 100

bytes with error: The server experienced an unexpected error when

processing the request (org.apache.kafka.clients.producer.internals.

ErrorLoggingCallback)

java: target/snappy-1.1.1/snappy.cc:423: char* snappy::internal::

CompressFragment(const char*, size_t, char*, snappy::uint16*, int): Assertion

`0 == memcmp(base, candidate, matched)' failed.

errors on publishing large amount of messages

Snappy

failed on distributed data

single

multiple

topics

Snappy

small network footprint

Hadoop friendly

easy schema verification

Kafka Offset Monitor

Improve security

Kafka-1682

Kafka <= 0.8.2

No security !

Kafka > 0.8.2

unix-like users, permissions, ACL

Manage your topics

cz.geecon.demo.basicGroup Topic

Improved securityAuthentication and authorization interfaces provided

By Default:

You can create any topic in your group

You can publish everywhere (in progress)

Group owner defines subscriptions

Think ahead

Consumer backoff

You can’t have exactly one delivery

http://bravenewgeek.com/you-cannot-have-exactly-once-delivery/

Improved offset managementHermesProducer

Hermes consumer

Publish event

Commited

Local unsent events

Read event

Improved offset management

Hermes consumer

Local unsent events

New event

Serviceinstance

Hermes consumer

Local unsent events

New event

Serviceinstance

HTTP 503 Unavailable

Local unsent events

New event

Serviceinstance

HTTP 503 Unavailable

Check TTL & Add to queue

Hermes consumer

Consumer backoff

100% adapt 1/s 1/min

Turn back the time

PUT /groups/{group}/topics/{topic}/subscriptions/{subscription}/retransmission -8h

Blog: http://allegro.tech

Twitter: @allegrotechblog

Twitter: @debskichris

Geecon.cz 2015 debski krzysztof

Technology

Krzysztof Kamil Baczyński

KRZYSZTOF KUNERT

mjr Krzysztof Gaczyński

Dr Krzysztof Koc

Krzysztof kieslowski

Krzysztof Penderecki

Krzysztof Głomb

Krzysztof Hetman Mar Krzysztof Babisz Ewa Dumkiewicz-Sprawka Kurier …mdk2.lublin.pl/debiuty/debiuty2014/DEBIUTY_2014_OPIS.pdf · 2015. 2. 10. · Kurier Lubelski nasze miasto.pl

Krzysztof Osiak, Krzysztof Wrzeszcz - Gemalto, Infostream - Bezpieczeństwo tożsamości oraz informacji klientów

Krzysztof Musiał

Krzysztof Kowalski

Krzysztof BANEK

KRZYSZTOF KAMIL BACZYŃSKI

Krzysztof Rumanowski

Krzysztof Turzański

krzysztof bojarczuk_2005

Krzysztof Wodiczko

Krzysztof Juszczyszyn krzysztof@pwr.wroc.pl ii.pwr.wroc.pl/~juszczyszyn

Lukasz Debski: Kačky a žvýkačky

Krzysztof Jażdżewski