[오픈소스컨설팅] Messaging Benchmark

주식회사 오픈소스컨설팅

Messaging Test

- Ø MQ, RabbitMQ, SQS, Kafka, etc -

2 - Internal Use Only -

Basic Process

Database/NoSQL

Gateway Module API Module

Web Apps Function

Component 1

Service Layer

Business Layer

Data Access Layer

Component 2

Service Layer

Business Layer

Data Access Layer

API Access

2. Put

1. API call (Servlet Param)

…

Messaging or Task

3. Reply(JSON)

4. Deliver Msg to Component

Client, CP/SP

Queue

Description Component

API Module • Polling data, data parsing • Send parsed data to message queue

Gateway Module • Subscribe from message queue, retrieve room data from data stores • Send data to Push Server

• Consumer pool


Kafka Performance Test – 2.8 Ghz, 4core, 8GB memory

Producer Batch-size 200 [jboss@localhost kafka-0.7.0]$ ./bin/kafka-producer-perf-test.sh --brokerinfo broker.list=0:localhost:9092 --messages 100000 --topic test --threads 1 --message-size 2048 [2012-05-28 09:02:31,236] INFO Creating sync producer for broker id = 0 at localhost:9092 (kafka.producer.ProducerPool) [2012-05-28 09:02:37,730] INFO Disconnecting from localhost:9092 (kafka.producer.SyncProducer) [2012-05-28 09:02:37,733] INFO Total Num Messages: 100000 bytes: 204800000 in 6.567 secs (kafka.tools.ProducerPerformance$) [2012-05-28 09:02:37,742] INFO Messages/sec: 15227.6534 (kafka.tools.ProducerPerformance$) [2012-05-28 09:02:37,742] INFO MB/sec: 29.7415 (kafka.tools.ProducerPerformance$) [jboss@localhost kafka-0.7.0]$

Consumer [2012-05-28 09:02:49,552] INFO Consumer test-consumer-group_localhost.localdomain-1338210169298-69d7d792 selected partitions : test:0-0: fetched offset = 2102058861: consumed offset = 2102058861 (kafka.consumer.ZookeeperConsumerConnector) [2012-05-28 09:02:49,558] INFO end rebalancing consumer test-consumer-group_localhost.localdomain-1338210169298-69d7d792 try #0 (kafka.consumer.ZookeeperConsumerConnector) [2012-05-28 09:02:49,560] INFO Sleeping for 5 seconds. (kafka.tools.ConsumerPerformance$) [2012-05-28 09:02:49,561] INFO FetchRunnable-0 start fetching topic: test part: 0 offset: 2102058861 from 127.0.0.1:9092 (kafka.consumer.FetcherRunnable) [2012-05-28 09:02:54,561] INFO starting threads (kafka.tools.ConsumerPerformance$) [2012-05-28 09:03:20,065] INFO thread[7], nMsgs:100000 bytes:204800000 nMsgs/sec:3923.57 MB/sec:7.66 (kafka.tools.ConsumerPerformance$) [2012-05-28 09:03:33,065] INFO thread[9], nMsgs:0 bytes:0 nMsgs/sec:0.00 MB/sec:0.00 (kafka.tools.ConsumerPerformance$) [2012-05-28 09:03:33,067] INFO thread[8], nMsgs:0 bytes:0 nMsgs/sec:0.00 MB/sec:0.00 (kafka.tools.ConsumerPerformance$) [2012-05-28 09:03:33,070] INFO thread[7], nMsgs:100000 bytes:204800000 nMsgs/sec:2597.47 MB/sec:5.07 (kafka.tools.ConsumerPerformance$) [2012-05-28 09:03:33,073] INFO thread[6], nMsgs:0 bytes:0 nMsgs/sec:0.00 MB/sec:0.00 (kafka.tools.ConsumerPerformance$) [2012-05-28 09:03:33,075] INFO thread[5], nMsgs:0 bytes:0 nMsgs/sec:0.00 MB/sec:0.00 (kafka.tools.ConsumerPerformance$) [2012-05-28 09:03:33,077] INFO thread[4], nMsgs:0 bytes:0 nMsgs/sec:0.00 MB/sec:0.00 (kafka.tools.ConsumerPerformance$) [2012-05-28 09:03:33,078] INFO thread[3], nMsgs:0 bytes:0 nMsgs/sec:0.00 MB/sec:0.00 (kafka.tools.ConsumerPerformance$)

Description Location

Producer • Batch-size가 1일 경우 상당한 퍼포먼스 문제 • 위의 테스트는 batch-size 200으로 처리(200개 메시지 들어올 경우 flush)

Consumer • Multi consumer 일 경우라 하더라도 하나의 파티션에 대해 하나의 consumer만 처리 • Consumer pool이 하나의 디스크로 인하여 의미가 없음

• Consumer pool

Producer Batch-size 1 [jboss@localhost kafka-0.7.0]$ ./bin/kafka-producer-perf-test.sh --brokerinfo broker.list=0:localhost:9092 --messages 100000 --topic test --threads 1 --message-size 2048 --batch-size 1 [2012-05-28 09:25:10,494] INFO Closing all sync producers (kafka.producer.ProducerPool) [2012-05-28 09:25:10,494] INFO Disconnecting from 127.0.0.1:9092 (kafka.producer.SyncProducer) [2012-05-28 09:25:10,510] INFO thread 2: 10000 messages sent 1682.3688 nMsg/sec 0.1604 MBs/sec (kafka.tools.ProducerPerfor mance$SyncProducerThread) [2012-05-28 09:25:10,510] INFO Closing all sync producers (kafka.producer.ProducerPool) [2012-05-28 09:25:10,510] INFO Disconnecting from 127.0.0.1:9092 (kafka.producer.SyncProducer) [2012-05-28 09:25:10,512] INFO thread 3: 10000 messages sent 1769.9115 nMsg/sec 0.1688 MBs/sec (kafka.tools.ProducerPerfor mance$SyncProducerThread)

http://www.mail-archive.com/[email protected]/msg01016.html







Rabbit MQ, HornetQ – 4core, 8GB Memory

Producer /Consumer Simultaneously [jboss@localhost rabbitmq-java-client-bin-2.8.2] ./runjava.sh com.rabbitmq.examples.MulticastMain --size 2048 starting consumer #0 starting producer #0 time: 6.000s, sent: 7198 msg/s, received: 7190 msg/s, min/avg/max latency: 1463/3982/23183 microseconds time: 7.000s, sent: 7335 msg/s, received: 7341 msg/s, min/avg/max latency: 1508/2892/6231 microseconds time: 8.000s, sent: 7386 msg/s, received: 7370 msg/s, min/avg/max latency: 1358/3691/15787 microseconds time: 9.000s, sent: 7262 msg/s, received: 7270 msg/s, min/avg/max latency: 1306/3091/7463 microseconds time: 10.000s, sent: 7283 msg/s, received: 7287 msg/s, min/avg/max latency: 1551/2924/7089 microseconds time: 11.000s, sent: 7164 msg/s, received: 7170 msg/s, min/avg/max latency: 1565/3522/11979 microseconds time: 12.000s, sent: 6043 msg/s, received: 6034 msg/s, min/avg/max latency: 1097/8846/41617 microseconds time: 13.000s, sent: 6040 msg/s, received: 6051 msg/s, min/avg/max latency: 943/11451/58797 microseconds [jboss@localhost rabbitmq-java-client-bin-2.8.2] ./runjava.sh com.rabbitmq.examples.MulticastMain --size 10240 starting consumer #0 starting producer #0 ime: 5.000s, sent: 2447 msg/s, received: 2447 msg/s, min/avg/max latency: 550/1344/17682 microseconds ime: 6.000s, sent: 2701 msg/s, received: 2702 msg/s, min/avg/max latency: 569/1134/4756 microseconds ime: 7.000s, sent: 2732 msg/s, received: 2731 msg/s, min/avg/max latency: 534/1122/3344 microseconds ime: 8.000s, sent: 2728 msg/s, received: 2730 msg/s, min/avg/max latency: 551/1121/2082 microseconds ime: 9.000s, sent: 2798 msg/s, received: 2798 msg/s, min/avg/max latency: 606/1116/2105 microseconds

[Rabbit MQ] [HornetQ, ActiveMQ]

Description Solution

RabbitMQ • Stable send/receive process • Can use AMQP(High performance) • Support runtime queue create/delete(very flexible)

HornetQ • High performance if message size is large • Support Linux AIO, JMS Bridge

• Embedded code 가능


Amazon SNS, SQS

Message send cost : avg 0.125 sec per message

Sender 1

Sender 2

Receiver 1

Receiver 2

[SQS Architecture]

Sender 1

Sender 2

Receiver 1

Receiver 2

Web Application Server

Endpoint URL Queue

[SNS Architecture]

Description Location

Producer • Sending part is HTTP client method call • Sending speed is very slow

• Call public DNS

Consumer • Processing time on receiver depends on web application server performance • Consumer is processed by web application server thread

• HTTP Thread


Comparison

Criteria Netty Kafka SNS

Short Response Time

Short Messaging Routes Short Short Long

Connection Handshake Cost Low (Connection Pooling Supported)

Middle (Handshakes on producers/consumers)

High (HTTP Connectionless Only)

Asynchronous Processing Async / Sync Supported Async Supported Async Supported

(HTTP Long Pooling)

Light-weight Software Stacks Light Light Heavy

High Availability No Single Point of Failures

SPF exists (Zookeeper Required)

HA Supported

(Zookeeper Embedded by default)

HA Supported

References Twitter, Plurk, Eucalyptus, Siemens

Linked-in, Tumblr (used for message store, NOT pub/sub)

AWS customers

Minimum System Complexity

Supported Data Formats Web Socket, Protobuf, POJO, Text, Avro, Thrift, Byte Array

Byte Array only Text only

Simple Object Model Simple Complex

(Kafka’s topic object model does not fit into the event model)

Simple

Minimum System Complexity

Easy Configuration / Hot-deployable /

Auto-scalable

Yes Yes Yes

Remarks Zookeeper required High-speed disk I/O required Callback web app required

Ø MQ Test Report


ØMQ Overview

Features Intelligent socket library for messaging

Many kinds of connection patterns

Multiplatform, multi-language (30+)

Fast (8M msg/sec, 30usec latency)

Small (20K lines of C++ code)

Open source LGPL (large community)

[Basic] [Forwarding]

[Pub/Sub] [Broadcast]


ØMQ Throughput – 4 core, 8GB Memory

[root@localhost perf]# ./local_thr.sh

tcp://eth0:5555 1024 100000

message size: 1024 [B]

message count: 100000

mean throughput: 39968[msg/s]

mean throughput: 327.417856[Mb/s]

1Kb Message [root@localhost perf]# ./remote_thr.sh

tcp://192.168.56.101:5555 1024 100000

Sent elapsed time : 2.339 sec

Sender Receiver


tcp://eth0:5555 2048 100000





2Kb Message [root@localhost perf]# ./remote_thr.sh

tcp://192.168.56.101:5555 2048 100000


4Kb Message


tcp://eth0:5555 4096 100000





[root@localhost perf]# ./remote_thr.sh

tcp://192.168.56.101:5555 4096 100000


10Kb Message


tcp://eth0:5555 10240 100000





[root@localhost perf]# ./remote_thr.sh

tcp://192.168.56.101:5555 10240 100000



Rabbit MQ Throughput – 4 core, 8GB Memory

[jboss@localhost rabbitmq-java-client-bin-2.8.2] ./runjava.sh com.rabbitmq.examples.MulticastMain --size 2048 starting consumer #0 starting producer #0 time: 6.000s, sent: 7198 msg/s, received: 7190 msg/s, min/avg/max latency: 1463/3982/23183 microseconds time: 7.000s, sent: 7335 msg/s, received: 7341 msg/s, min/avg/max latency: 1508/2892/6231 microseconds time: 8.000s, sent: 7386 msg/s, received: 7370 msg/s, min/avg/max latency: 1358/3691/15787 microseconds time: 9.000s, sent: 7262 msg/s, received: 7270 msg/s, min/avg/max latency: 1306/3091/7463 microseconds time: 10.000s, sent: 7283 msg/s, received: 7287 msg/s, min/avg/max latency: 1551/2924/7089 microseconds time: 11.000s, sent: 7164 msg/s, received: 7170 msg/s, min/avg/max latency: 1565/3522/11979 microseconds time: 12.000s, sent: 6043 msg/s, received: 6034 msg/s, min/avg/max latency: 1097/8846/41617 microseconds time: 13.000s, sent: 6040 msg/s, received: 6051 msg/s, min/avg/max latency: 943/11451/58797 microseconds [jboss@localhost rabbitmq-java-client-bin-2.8.2] ./runjava.sh com.rabbitmq.examples.MulticastMain --size 10240 starting consumer #0 starting producer #0 ime: 5.000s, sent: 2447 msg/s, received: 2447 msg/s, min/avg/max latency: 550/1344/17682 microseconds ime: 6.000s, sent: 2701 msg/s, received: 2702 msg/s, min/avg/max latency: 569/1134/4756 microseconds ime: 7.000s, sent: 2732 msg/s, received: 2731 msg/s, min/avg/max latency: 534/1122/3344 microseconds ime: 8.000s, sent: 2728 msg/s, received: 2730 msg/s, min/avg/max latency: 551/1121/2082 microseconds ime: 9.000s, sent: 2798 msg/s, received: 2798 msg/s, min/avg/max latency: 606/1116/2105 microseconds

[Rabbit MQ Result]

Description Solution

RabbitMQ

• Stable send/receive process • Can use AMQP(High performance) • Similar producer/consumer speed • Support runtime queue create/delete(very flexible)

Producer/Consumer Simultaneously; 2K and 10K message

Producer/Consumer


Comparison – RabbitMQ vs. ØMQ

Criteria RabbitMQ Ø MQ

Short Response Time

Short Messaging Routes Short Short

Performance

(Based on 10Kb) 2798 msg/sec

4.66 times faster than RabbitMQ – 13048 msg/sec

Asynchronous Processing

Sync, Async Supported Async based event processing

Light-weight Software Stacks

Light More Light

High Availability No Single Point of

Failures HA Cluster Support No Broker Architecture

Flexibility

Supported Data Formats Various(JSON, POJO, etc.) Various(JSON, Thrift, Google ProtoBuf, etc.)

Easy Configuration GUI Based Configuration Implementation needed

Runtime Configuration Runtime deploy for queue, topic Implementation needed

Commercial Support SpringSource(VMWare) iMatrix

Monitoring Many plugin for monitoring, web based GUI Implementation needed

Remarks

• Powerful web based monitoring

• AMQP1) Support

• Dynamic topic management

• Easy of Development

• High throughput/Low latency

• Auto reconnect among peer

• Run on arbitrary platforms (Windows, Android)

1) AMQP(Asynchronous Message Queuing Protocol) : Supported by Microsoft, Red Hat, VMware, Cisco, Novell, SoftwareAG, etc.

Ø MQ is very lightweight and fast, but we’ll have to implement most of features ourselves

Flexibility, reliability, and easy to use are important in this project, not performance

Appendix A. Ø MQ - Implementation Model -


Install

Ø MQ core is built using C language

Prerequisites : autoconf, automake, gcc, gcc-g++, make, git, java, libtool

[Documents]

ZeroMQ : http://www.zeromq.org/

ZeroMQ Java : http://www.zeromq.org/bindings:java

Guide : http://zguide.zeromq.org/page:all

Korean : http://kr.zeromq.org/

[ZeroMQ Core Install]

wget http://download.zeromq.org/zeromq-2.2.0.tar.gz

tar -zxvf zeromq-2.2.0.tar.gz

cd zeromq-2.2.0

./configure --with-pgm Multicast support

make

sudo make install

[ZeroMQ Java Install]

git clone https://github.com/zeromq/jzmq.git

cd jzmq

./autogen.sh

./configure

make

sudo make install


Client/Server

import org.zeromq.ZMQ;

public class hwclient {

public static void main (String[] args){

ZMQ.Context context = ZMQ.context (1);

ZMQ.Socket socket = context.socket (ZMQ.REQ);

socket.connect ("tcp://localhost:5555");

socket.send ("Hello", 0);

System.out.println (socket.recv(0));

}

}

import org.zeromq.ZMQ;

public class hwserver {

public static void main (String[] args) {

ZMQ.Context context = ZMQ.context(1);

ZMQ.Socket socket = context.socket(ZMQ.REP);

socket.bind ("tcp://*:5555");

while (true) {

byte [] request = socket.recv (0);

socket.send("World", 0);

}

}

}


Queue Load Balancing

import zmq

context = zmq.Context()

socket = context.socket(zmq.REQ)

socket.connect("tcp://127.0.0.1:5000")


for i in range(10):

msg = "msg %s" % i

socket.send(msg)

print "Sending", msg

msg_in = socket.recv()

import zmq


socket = context.socket(zmq.REP)

socket.bind("tcp://127.0.0.1:5000")

while True:

msg = socket.recv()

print "Got", msg

socket.send(msg)

import zmq


socket = context.socket(zmq.REP)

socket.bind("tcp://127.0.0.1:6000")

while True:

msg = socket.recv()

print "Got", msg

socket.send(msg)

Each server will be received five messages equivalently


Broadcast - Pub/Sub

import zmq

from random import choice


socket = context.socket(zmq.PUB)

socket.bind("tcp://127.0.0.1:5000")

countries = ['netherlands','brazil','germany','portugal']

events = ['yellow card', 'red card', 'goal', 'corner', 'foul']

while True:

msg = choice( countries ) +" "+ choice( events )

print "->",msg

socket.send( msg )

import zmq


socket = context.socket(zmq.SUB)


socket.setsockopt(zmq.SUBSCRIBE, "netherlands")

socket.setsockopt(zmq.SUBSCRIBE, "germany")

while True:

print socket.recv()

Subscriber will be received only “netherlands” and “germany”


OPEN

SHARE

CONTRIBUTE

ADOPT

REUSE


Technology

[오픈소스컨설팅] Messaging Benchmark