Pika driver

Copyright © 2015 Mirantis, Inc. All rights reserved

www.mirantis.com

Pika RabbitMQ driver

Design and features overview

Dmitriy UkhlovSoftware Engineer

[email protected], launchpad: dukhlov


Pika driver requirements

● should be developed from scratch

● should be developed in optimal way regarding to all Pika client library new features

and best practices;

● should be fully compatible with current driver interface;

● may change internals and does not guarantee compatibility with Kombu driver

(means you can not use old Kombu driver for some set of services and new Pika

driver for other services)

● should support only current actual features without any deprecated features.


Oslo.messaging driver use cases

1. CALL – client sends message to one of the set of servers defined

by target which currently listen topic and waits for response

during short timeout;

2. CAST – client sends message to each or only one of the set of

servers defined by target which currently listen topic;

3. NOTIFY – client sends message to each of the set of servers

which have permanent subscription for topic and will receive

message even if they don't listen topic currently


CALL solution

Exchangename =

rpc_exchangetype = directauto_delete =

Truedurable = False

routing_key = <topic> or <topic>.<server>

reply_q = <project>.<prog_name>.<process_id>

expiration = <timeout>mandatory = Truedelivery_confirmation = enabledpersistence = False

Queuename =<topic>auto_delete =

Falsedurable = FalseX-expires = 60 s

Queuename =

<topic>.<server>auto_delete = False

durable = FalseX-expires = 60 s

<server>

Exchangename =

rpc_reply_exchange

type = directauto_delete =

Truedurable = False

Queuename

=<project>.<prog_name>.<process_id>auto_delete = False


routing_key = <project>.<prog_name>.<process_id>expiration = <timeout>mandatory = Falsedelivery_confirmation = enabledpersistence = False


CALL solution features

● Expired RabbitMQ resources (queues, exchanges and messages) cleanup;

● Using direct exchanges to make routing faster;

● Using single reply listener per client process to make call faster;

● Using delivery confirmation to get maximum information about connectivity

problems;

● Connectivity fault tolerant (call will survive short time connectivity problem)

● Using message listening without confirmation (prevents redelivery and processing

one message a few times)


CAST solution

Exchangename =

openstack_rpctype = directauto_delete =

Truedurable = False

routing_key = <topic> or <topic>.<server>


Queuename =<topic>auto_delete =

Falsedurable = FalseX-expires = 60 s

Queuename =

<topic>.<server>auto_delete = False


<server>

Exchangename =

openstack_rpc_fanout_<topic>

type = fanoutauto_delete = True

durable = False



CAST solution features

● Expired RabbitMQ resources (queues, exchanges and messages) cleanup;

● Using direct and fanout exchanges (instead topic) to make routing faster;

● Using delivery confirmation to get maximum information about connectivity

problems;

● Connectivity fault tolerant (detecting problem which could be recovered and retry)

● Using message listening without confirmation (prevents re-delivery and processing

one message a few times)


Notify solution

Exchangename =

openstack_notificationtype = direct

auto_delete = Falsedurable = True

routing_key = <topic>.<priority>

expiration = <timeout>mandatory = Truedelivery_confirmation = enabledpersistent = <notification_persistence>

Queuename

=<topic>.<priority>auto_delete = False

durable = True

<primary server><secondary server>

Queuename =<custom_server_name>

auto_delete = Falsedurable = True


Notify solution features

● Durable exchanges and queues are used to survive RabbitMQ restart● Client will create default queue <topic>.<priority> if it can not deliver message to

any other queue;● By default delivery is guaranteed if you use only one listener and it listen default

queues. In any other case you prepare RabbitMQ durable queues, exchanges and its bindings before starting Notification clients (for example by running all servers before client)

● Using direct exchange (instead of topic) to make routing faster;● Using delivery confirmation to get maximum information about connectivity

problems;● Connectivity fault tolerant (detecting problem which could be recovered and retry)● Using message listening with confirmation (prevents message losing in case of

notification listener crash)


Key changes in pika driver comparing to kombu

● using direct exchanges instead of topic exchanges● using separate exchanges for different message types● using optional persistence only for notifications (durable exchange and queues and persistent

messages)● rpc queues and exchanges with expiration period - will we deleted by RabbitMQ if any service does

not connected.● new approach of creating queues for notifications by client (client tries to send notification with

mandatory flag and declare default queue it message get returned, which means that there is now queue for routing message now)

● using single reply listener per application (not per request)● using own reconnection mechanism implemented in pika driver and able to work correctly in ha

mode● HA configuration is removed. It could be configured via RabbitMQ policies● using heartbeat mechanism implemented in pika library

Presentations & Public Speaking

Pika driver