Upload
lyo-kato
View
1.179
Download
2
Embed Size (px)
DESCRIPTION
Perl Oceanを作成したときの資料 XMPP/Jingleなどのプロトコルの説明や、古いブラウザやWebSocketのような新技術とのインテグレーションについて
Citation preview
XMPP Based Realtime Communication
Framework Suite
!
加藤 亮 2012/03
Ocean
Realtime
• Data (micro contents)
• Delivery Method ( Push )
• UI
PUSH Friendly Contents
• Notification
• Activity Stream
• Message, Chat
• Presence
PUSH
• Long Polling
• XMPP
• HTML5 (SSE, WebSocekt)
• Smart phone (APNS, C2DM)
UI
• Ajaxify - auto view updating
Ocean
Framework Suite
• Front Server
• Cluster
• HTTP Binding
• P2P(Audio/Video chat) 予定
Protocol
• XMPP Based
• XMPPのサブセットをサポート
• XMPPと内部で互換性を持たせた数種類の独自プロトコル
• External WebSocket Binding
• HTTP Binding ( WebSocket/SSE/XHR )
Jabber/XMPP
examples
• google talk
• facebook chat
• PSN
• iMessage
• google wave ( S2S )
Stream Procedure
• Stream Initialization
• TLS Negotiation
• Authentication
• Resource Binding
• Session Establishment
• Roster Request
• Initial Presence
Stanza
• Feature
• Auth
• Message
• Presence
• IQ
ID
• JID (example: [email protected]/resource )
• Full JID ([email protected]/resource)
• Bare JID ([email protected])
Authentication• SASL
• DIGEST-MD5
• PLAIN ( under TLS )
• EXTERNAL ( for S2S TLS )
• We need
• OAuth 2.0?
• Cookie/Ticket ( for WebSocket )
Message
Message
• service notification (type=headline)
• chat message (type=chat)
• group chat message (type=groupchat)
• normal ? message (type=message)
Presence
Presence
• presence
• subscription
• Ocean doesn’t support subscription
• extension
• join/leave a room
Presence
• show (chat, away, dnd, xa )
• status (custom message)
• priority
Presence traps
• protocol design
• presence:connection = 1:1
• de facto usage
• presence:user = 1:1
Presence De Facto Usage
• one presence for one user
• we don’t need priority
• ‘chat’ and ‘away’ are enough for ‘show’ param
Resource Binding
• node@domain/resource
• you can choose ‘home’ or ‘work’
• de facto usage
• users and client softwares don’t mind what the resource is.
• so server generates random string (like a cookie )
Session Establishment
• Historical Reason
• Do nothing
IQ
• Roster ( Friend List )
• vCard ( Profile )
• bind, session, ping, disco, etc...
• packets should be back to sender
Roster & vCard• like OpenSocial’s ‘People Data API’
Roster
• there isn’t profile image, which is included in vCard
• no pagination control by default
vCard traps
• just for profile image
• but base64 encoded binary
• we want URL
Subscription & Roster Push
• Friend Request
• Relationship State Control
C2S & S2S
• Client to Server
• Server to Server
• Ocean doesn’t support S2S
C2S Example
gmail
User A User B
C2S StreamC2S Stream
S2S Example
gmail.com
User A User B
chat.facebook.com
User DUser C
S2S Stream
S2S Problems
• We have to manage new database that represents ‘relationships’ between users of our service and ones of other service.
• We have to manage a white list or spam control. ( like OpenID )
• Authentication Compatibility Cost
• Scaling Cost
• Error Handling
Implementation
Other Implementations• I/O Multiplexing
• Node.js
• Apache MINA, naggati
• Jabber/XMPP software
• ejebberd
• DJabberd
• Openfire
Ocean’s XMPP support
• subset
• customized
• fixed features(not pluggable )
• assumed cluster installation
Why subset?
• Ocean is just for Web-based Social Network Service
• Create/Modify friendships on the web site.
• S2S costs too much
• Developers have to write only minimum number of event handlers.
Protocol Customized
• resource binding & roster/vCard responses support extra-attributes.
• Because of regardless of HTTP federation, especially URL
• WebSocket binding (experimental )
• try demo application
• Simple-MUC
Not flexible and pluggable?
• It may cause complication. requires detailed protocol knowledge, framework knowledge.
• When the servers are distributed, front-server-plugins require complicated spec. See ‘Cluster’ section.
• There are de-facto-standard specs.
Why full scratch?
• Proprietary Reason
• We may need non-standard features
• Rapid Updating (community based development requires many arguments)
• Architectural Dependencies
• No Money
Inside Ocean
Getting Started
• All you need to do is ...
• generate project template
• edit configuration file
• prepare your handler class
Generate Project Template
• ./ocean
• cd Foobar
Configuration
• vi conf/ocean.yml
Write Handler Class
• inherit Ocean::Handler
• implement all event handler methods
Ocean Components
Server Stream Manager
Stream Stream Stream
Handler
Daemonizer Signal Handler Listener Stream Builder
Stream Components
IO Components
Inside Stream
Stream Manager
Encoder
Decoder
Socket
Protocol States
Client
SASL
Active
...
ServerHandler
Events
Problem?
• Bad performance
• too many call stack on each event
• each component consume memory
Benefit
• Testability
• You can switch to stub components
• Customizability
• You can switch Encoder/Decoder
Pay attention to memory usage
• 4byte on a Stream object
• 4x10,000byte on a Server witch has 10,000 connection
• bless Array instead of Hash
• use Config instance as singleton object
• use delegate object for event handling
Your Service
Databases
Handler
Handler
Users RelationshipsConnection Info
User A Home chat
User A Work away
User B Home chat
User C Home chat
Files
Profile Pictures
Handler uses 2 or 3 kinds of store
• Persistent Data Store (users, relationships)
• Half Persistent Data Store. Same lifecycle as XMPP service (connection map)
• Cache system ( if needed )
Handler Implementation
Handler Example
• modules under Ocean::Standalone::Handler namespace are good example
Server Events
• on_node_init
• on_node_exit
Stream Events
• on_sasl_auth_request
Stream Events
• on_bind_request
Stream Events
• on_message
Stream Events
• on_presence
• on_initial_presence
• on_unavailable_presence
Presence doesn’t scale• user_001 login-ed ( send initial-presence )
• get all friends of user_001
• get all available-resources of the friends
• send user_001’s presence to all the available-resources.
• send presences of the all available-resources to user_001
It’s important to limit the maximum number of friends.
Stream Events
• on_silent_disconnection
Stream Events
• on_oster_request
Stream Events
• on_vcard_request
Stream Events
• on_too_many_auth_attempt
Start & Stop
• bin/ocean-start
• bin/ocean-start --daemonize
• bin/ocean-stop ( if daemonized )
Cluster
• C10K Problem on Persistent Connection
• Blocking IO multiplexing
• Message delivery/Presence broadcast
Message Delivery
XMPP Server 1
User A (Home)
User B (Home)
XMPP Server 2
User C (Home)
Delivery Server
Connection Map
User A Home 1
User A Work 2
User B Home 1
User C Home 2
User A (Work)
Use 2 kinds of gearman
• Background Job Dispatcher
• Inbox ( 1 inbox for each front-server )
Message Delivery
XMPP Server 1 XMPP Server 2
Delivery Server
Connection Map
User A Home 1
User A Work 2
User B Home 1
User C Home 2
Background Job Dispatcher (gearmand)
inbox1 (gearmand)
inbox2 (gearmand)
Message Delivery
XMPP Server 1 XMPP Server 2
Delivery Server
Connection Map
User A Home 1
User A Work 2
User B Home 1
User C Home 2
Background Job Dispatcher (gearmand)
inbox1 (gearmand)
inbox2 (gearmand)
Message Delivery
XMPP Server 1 XMPP Server 2
Delivery Server
Connection Map
User A Home 1
User A Work 2
User B Home 1
User C Home 2
Background Job Dispatcher (gearmand)
inbox1 (gearmand)
inbox2 (gearmand)
Connection Map• too many ‘read’
• ‘write’ on resource-binding, presence-change
• Implementation
• TokyoTyrant
• HandlerSocket
• must not be volatile
Ocean::Cluster
• Ocean::Cluster::Gearman::Server
• Ocean::Cluster::Gearman::Worker
Cluster support on front server
• set config ‘handler_class=Ocean::Cluster::Gearman::Server::Handler’
• set inbox and background-job-dispatcher
Front server components
XMPP Server
inbox (gearmand)
Background Job Dispatcher (gearmand)
Ocean::Cluster::Gearman::Server::Handler
Build worker project
!
• same as front server
• build project-template
• edit configuration
• write your handler class
Generate Project Template
• ocean_cluster_gen --project=FooWorker --handler=FooWorker::Handler
• cd FooWorker
Configuration
• set information about all your inboxes
• set information about background-job-dispatcher
Worker
Worker side (delivery server) components
inbox0 (gearmand)
Background Job Dispatcher (gearmand)
inbox1 (gearmand)
Handler
Serializer
Subscriber Deliverer
Process Manager
Connection Map
User A Home 1
User A Work 2
User B Home 1
User C Home 2
DB (users, relationships, etc...)
Transaction
• 同じユーザーからのイベント順序が壊れないように調整が必要
Operation
Balancing
• Round-Robin
• DNS SRV Record
• _xmpp_client._tcp.chat.mixi.jp 86400 IN SRV 10 20 5222 chat1.mixi.jp
• _xmpp_client._tcp.chat.mixi.jp 86400 IN SRV 10 5 5222 chat2.mixi.jp
chat1.mixi.jp
User A (Home)
Service Status Store
name connections wait
chat1 100000 2
chat2 1000 1
chat3 100000 2
chat2.mixi.jp chat3.mixi.jp
Update Periodically
Status API
Service Status API and Client Hack
chat1.mixi.jp
User A (Home)
Service Status Store
name connections wait
chat1 100000 2
chat2 1000 1
chat3 100000 2
chat2.mixi.jp chat3.mixi.jp
Update Periodically
Status API
Service Status API and Client Hack
chat1.mixi.jp
User A (Home)
Service Status Store
name connections wait
chat1 100000 2
chat2 1000 1
chat3 100000 2
chat2.mixi.jp chat3.mixi.jp
Update Periodically
Status API
Service Status API and Client Hack
Update & Reboot
• Worker side handler includes service-code, so it’s required updating and rebooting frequently. Worker’s job is stateless. So, it’s no problem.
• Front-end server doesn’t include service-code. So you don’t need to reboot it frequently, but when you want to do it, you have to pay attention.
Shutdown• You can’t shutdown server without client
awareness.
• You have to write re-connection on client side code
• Problem: algorithm distribution case
• At graceful shutdown, unavailable presences are handled correctly
• Problem: at non-graceful shutdown
Front-server Broken
• フロントがぶっ壊れたときのために、ConnectionMapから、壊れたサーバーにつないでたユーザー全員のunavailable
presenceを処理するバッチを用意する必要がある
Front-server maintainance
• algorithm分散のフロントを定期再起動するときは、代替が先に必要
• group_id = get_group(user_id)
• host = get_host(group_id)
• 代替サーバーを準備し、group/host
マップを書き換えてから、shutdown
Service(Client-side) Strategy
Device/Platform
• PC Web ( HTTP Binding )
• PC Desktop App(AIR, Titanium, or Native)
• Smartphone
• Tablet
• Browser extension
Service
• 1to1 Messaging
• Room? (group chat)
• Voice/Video chat
• Push Notification
Ocean supports
• WebSocket hybi-draft-10 support
• HTTPBinding(WebSocket/SSE/XHR)
HTTP Binding
Methods
• Polling ( Ajax ) + Post API (+APNS|C2DM)
• Long Polling ( Ajax, Comet ) + Post API
• ServerSentEvent + Post API
• WebSocket
• TCP XMPP
‘Stateless’ problems
• Different way of connection management from ordinary persistent connection
• single user-session may open many tabs
• user moves in short time. too many re-connection
• how to detect user to leave? (leave or move)
server 1
User A
server 2 server 3
Connection Map
User A Home 1
User A Work 2
User B Home 1
User C Home 2
too many ‘write’ and ‘presence’ in short time
open many tabs or hop http-link in short time
TabTab
Tab
mod_proxy(_balancer)
User B
when user leaves?
Solution 1 Algorithm Based Distribution
• ユーザー毎の宛先を固定することによって、Connection Mapを利用しなくても、特定のユーザーにパケットを届けることは可能になる
• ユーザーがWebサービスを開いてるかどうかは関係なくなる
Problem
• Connection Mapを利用しないと、正確なプレゼンスを把握できなくなる
Solution 2. Stream Aggregation
• 同一cookieを持つ接続をまとめあげるSessionオブジェクトを利用する
• 状態管理はSessionオブジェクトで工夫
• Pending TimerでLeaveの検知
User B
hb1.mixi.jp hb2.mixi.jp hb3.mixi.jp
User A
TabTab
TabUser C
TabTab
Tab
Connection Map
User A Home 1
User A Work 2
User B Home 1
User C Home 2Aggregate
connections as single session
algorithm based distribution: server_node=algorithm(user_id)
Manager Manager Manager
use timer to detect users to leave.
(*) number of simultaneous Ajax request directed to same domain is limited, so we need many nickname for each hb servers. (if we adopt long polling)
until the expiration, server should buffer messages if received
Server Stream Manager
User A Stream
User A Stream
Handler
Session
Handle Multiple Streams as One Single Session
User B Stream
User B Stream
Session
Auth
ResourceBinding
SessionEstablishment
InitialPresence
Handshake
Tasks
• We need to
• prepare social ad system
• improve UI (moveless)
PubSub Support
• XEP-0061 Publish Subscribe
• We don’t need full-spec support
• don’t allow use to publish/subscribe
• server publish event
Publish Event from Web Service
XMPP Server 1
User A (Home)
User B (Home)
XMPP Server 2
User C (Home)
Delivery Server
User A (Work)
Web ServiceUserEvents
(Notification, Activity)
XMPP (HTTPBinding) Streams
Web Service
UserEvents (Notification,
Activity)
APNSC2DM EmailWebHook PubsubHubbub
DeliveryCondition: ContentKind, UserPresence
Groupchat
Overview
• XMPP has MUC extension spec. but it’s over spec, so we need arrangement
• Costs too much like presence
• limit numbers of channels and participants
• Static-type groupchat should keep synchronicity with ‘group(room?)’ feature on our service
Video/Voice Chat
Problem?
• Isn’t Identity Association enough?
• AccountManage (android)
• FaceTime (iOS)
VoIP
Signaling Server
Peer A Peer B
Signaling Channel Signaling Channel
RTP
Data Channel(Audio, Video)
VoIP
Signaling Server
Peer A Peer B
RTP
NATs NATs
Cant’ get PeerB’s Transport Address
Cant’ get PeerA’s Transport Address
NAT Traversal
Hole Punching
Local Host
NATs
1. Local host connects to an external host
Hole Punching
Local Host
NATs
1. Local host connects to an external host
2. NAT allocates a transport address to localhost
192.168.0.1:10001
10.100.100.100:10002
MAPPING
Hole Punching
Local Host
NATs
1. Local host connects to an external host
2. NAT allocates a transport address to localhost
192.168.0.1:10001
10.100.100.100:10002
MAPPING
Hole Punching
Local Host
NATs
1. Local host connects to an external host
2. NAT allocates a transport address to localhost
3. An external host can send packet to local host through mapped global address
But every NAT doesn’t work as expected
STUN
STUN Server
STUN Client
NATs
Server Reflexive Transport Address
Host Transport Address
STUN
Signaling Server
Peer A Peer B
NATs NATs
STUN Server
STUN Server
Do Hole Punching And
Obtain Reflexive Address
STUN
Signaling Server
Peer A Peer B
NATs NATs
STUN Server
STUN Server
Exchange Obtained Reflexive Address
Through Signaling Server
STUN
Signaling Server
Peer A Peer B
NATs NATs
STUN Server
STUN Server
Direct Communication Path
TURN
Signaling Server
Peer A Peer B
NATs NATsTURN Server
relay packet
TURN
TURN Server
TURN Client
NATs
Remote Peer
Relayed Transport Address
Server Reflexive Transport Address
Host Transport Address
NATs
TURN
TURN Server
TURN Client
NATs
Remote Peer
Relayed Transport Address
Server Reflexive Transport Address
NATs
Allocation
Client ReflexiveTransport Relayed Transport
XXX.XXX.XXX.XXX:10001 YYY.YYY.YYY.YYY:10001
XXX.XXX.XXX.XXX:10001 YYY.YYY.YYY.YYY:10002
XXX.XXX.XXX.XXX:10002 YYY.YYY.YYY.YYY:10003
TURN
TURN Server
TURN Client
NATs
Remote Peer A
Relayed Transport Address
NATs
Remote Peer B
NATs
Remote Peer C
NATs
TURN
TURN Server
TURN Client
NATs
Remote Peer
Channel Binding
Client ReflexiveTransport Number Relayed Transport
XXX.XXX.XXX.XXX:10001 0x4001 YYY.YYY.YYY.YYY:10001
XXX.XXX.XXX.XXX:10001 0x4002 YYY.YYY.YYY.YYY:10002
XXX.XXX.XXX.XXX:10002 0x5001 YYY.YYY.YYY.YYY:10003
NATs
TODO: need peer reflexive address?
XMPP Jingle
• Signaling on XMPP
Serverless Messaging
Protocols
• UPnP
• Bonjour(Rendezvous)
Bonjour
• mDNS
• DNS-SD
Find Friends
• search service by DNS PTR records
• search host for indicated service by SRV records
• search IP address by A record
• (TXT records for capability and status)
TODO
• backward compatibilityの確保(iChat, etc)
• 現在の要件への現実的な対応(HTTPBinding)
• 将来の準備(WebRTC)