Embed Size (px)
DESCRIPTION
Agenda ● Variety of sync mechanisms ● Polling & Push ● RESTful API Design ● API Blueprint
Citation preview
How to build a scalable SNS via Polling & Push
Kewang
三竹資訊
2
Who I am
● 王慕羣● Java / Node.js / AngularJS● SQL-like / HBase
● Mentor: NIU CSIE
GitHub: kewangtw
Google Play: kewang
Facebook: kewangtw
Linkedin: kewangtw
Slideshare: kewang
Mail: [email protected]
Who Mitake is
三竹資訊
Who Mitake is
三竹資訊
大家都唸 Mitake
Who Mitake is
三竹資訊
大家都唸 Mitake ，但我們公司都唸 Mitake
Who Mitake is
三竹資訊
Mitake 不唸作 MiTAC 啊！！！
Who Mitake is
三竹資訊
Who Mitake is
三竹資訊● 簡訊平台
Who Mitake is
三竹資訊● 簡訊平台● 行動下單：
Who Mitake is
三竹資訊● 簡訊平台● 行動下單：
Who Mitake is
三竹資訊● 簡訊平台● 行動下單：不計其數
Who Mitake is
三竹資訊● 簡訊平台● 行動下單：不計其數● 行動銀行：
Who Mitake is
三竹資訊● 簡訊平台● 行動下單：不計其數● 行動銀行：臺銀、土銀、富邦、台新、聯邦、臺企銀、遠銀、華南、澳盛、郵局、合庫、渣打 ...等 18 家
Who Mitake is
三竹資訊● 簡訊平台● 行動下單：不計其數● 行動銀行：臺銀、土銀、富邦、台新、聯邦、臺企銀、遠銀、華南、澳盛、郵局、合庫、渣打 ...等 18 家● 產壽險：
Who Mitake is
三竹資訊● 簡訊平台● 行動下單：不計其數● 行動銀行：臺銀、土銀、富邦、台新、聯邦、臺企銀、遠銀、華南、澳盛、郵局、合庫、渣打 ...等 18 家● 產壽險：全球、明台、新光、新安東京、富邦 ...等
Who Mitake is
三竹資訊● 簡訊平台● 行動下單：不計其數● 行動銀行：臺銀、土銀、富邦、台新、聯邦、臺企銀、遠銀、華南、澳盛、郵局、合庫、渣打 ... 等
18 家● 產壽險：全球、明台、新光、新安東京、富邦 ... 等● 其他：
Who Mitake is
三竹資訊● 簡訊平台● 行動下單：不計其數● 行動銀行：臺銀、土銀、富邦、台新、聯邦、臺企銀、遠銀、華南、澳盛、郵局、合庫、渣打 ... 等 18家● 產壽險：全球、明台、新光、新安東京、富邦 ... 等● 其他： udn 買東西、手機逛週年慶、財政園地、證交所、綜所稅申報 ... 等
18
System Architecture
19
System Architecture
20
System Architecture (Backend)
21
System Architecture (Backend)
HadoopCon 2014
22
System Architecture (Frontend)
23
Agenda
● Variety of sync mechanisms● Polling & Push● RESTful API Design● API Blueprint
24
Variety of sync mechanisms
25
Variety of sync mechanisms
● Polling● Comet● Long Polling● WebSocket● Engine.IO (Socket.IO)
26
Pollingsend scheduling request
27
Polling
client server
28
Polling
client server
T0T1
T2
29
Polling
client server
T0T1
T2
T4T5
T6
30
Polling
client server
T0T1
T2
T4T5
T6
T8T9
T10
31
Polling
● Pros– Easy to implement
● Cons– No efficiency
32
Cometa never died HTTP request
33
Comet
client server
34
Comet
client server
T0T1
35
Comet
client server
T0T1
36
Comet
client server
T0T1
T3T2
37
Comet
client server
T0T1
T3
T6
T2
T5
38
Comet
client server
T0T1
T3
T6T8
T2
T5T7
39
Comet
client server
T0T1
T3
T6T8
T12
T2
T5T7
T11
40
Comet
● Pros– Save more network traffic
● Cons– Blocking IO issue– Always get server response, can't send another request
41
Long Pollingsend a long time request repeatedly
when response received
42
Long Polling
client server
43
Long Polling
client server
T0T1
44
Long Polling
client server
T0T1
45
Long Polling
client server
T0T1
T4
T3
46
Long Polling
client server
T0T1
T4
T3
T5
47
Long Polling
client server
T0T1
T4
T3
T5
48
Long Polling
client server
T0T1
T4
T3
T5
T21T20
49
Long Polling
● Pros– Save a little network traffic– Can send another request
● Cons– I don't know
50
WebSocketFDX communications channels over a single TCP connection
51
WebSocket
client server
52
WebSocket
client server
53
WebSocket
client server
T0T1
T2
54
WebSocket
client server
T0T1
T2
T3T4
T5
55
WebSocket
client server
T0T1
T2
T3T4
T5T9
T10T11
56
WebSocket
client server
T0T1
T2
T3T4
T5
T17T18
T19
T9T10
T11
57
WebSocket
● Pros– Bidirectional communication– Save more network traffic
● Cons– Proxy & Firewall issue
58
Engine.IOcommunication layer for Socket.IO
59
Engine.IO - Socket.IO history0.1~0.6.2 0.6.3~0.6.17 0.7.0~0.9.17 1.0.0~now
websocket ✓ ✓ ✓ ✓server-events ✓flashsocket ✓ ✓ ✓
htmlfile ✓ ✓xhr-multipart ✓ ✓xhr-polling ✓ ✓ ✓
jsonp-polling ✓polling ✓
60
Engine.IO - Overview
61
Engine.IO - Overview
● Ensure the most reliable realtime communication
62
Engine.IO - Overview
● Ensure the most reliable realtime communication● Always establishes a long-polling connection first
63
Engine.IO - Overview
● Ensure the most reliable realtime communication● Always establishes a long-polling connection first
– then tries to upgrade to better transports that are "tested" on the side
64
Engine.IO - Upgrade transport seamlessly
65
Engine.IO - Upgrade transport seamlessly
● Switches from polling to another transport in between polling cycles
66
Engine.IO - Upgrade transport seamlessly
● Switches from polling to another transport in between polling cycles
● To ensure no messages are lost, the upgrade packet will only be sent once all the buffers of the existing transport are flushed and the transport is considered paused
67
Engine.IO
client server
68
Engine.IO
client server
T0T1
69
Engine.IO
client server
T0T1
70
Engine.IO
client server
T0T1
71
Engine.IO
client server
T0T1T6
T5
T7
72
Engine.IO
client server
T0T1
T11
T6T5
T7T10
73
Engine.IO
client server
T0T1
T11
T6T5
T7T10
74
Engine.IO
client server
T0T1
T11
T6
T12T13
T14
T5
T7T10
75
Engine.IO
● Pros– Auto-switch different network scenario
● Cons– Use particular protocol at server & client
76
Polling & Push
77
Why we don't use WebSocket
● Server– Many corporate proxies block WebSocket traffic
● Client– Many personal firewalls and antivirus softwares block
WebSocket traffic
78
Push != Notification
79
Push != Notification
80
Push != NotificationPush Notification
Notification
Push Notification
Notification
81
Push - Overview
● Non-IM– Breaking news notification– Remittance notification– Call off work notification– Email notification– ......etc.
● IM– Message notification
82
How it works?
83
When unsafe APIs done - Server
1.Store affected devices' ID & API ID to database
2.Send Push to affected devices
84
When unsafe APIs done - Client
● Send an unified GET request– GET /datas
85
When unsafe APIs done - Client
● Send an unified GET request– GET /datas
● Push off– apply frequency rapidly to achieve realtime, but waste
resources
86
When unsafe APIs done - Client
● Send an unified GET request– GET /datas
● Push off– apply frequency rapidly to achieve realtime, but waste
resources● Push on
– apply frequency long to save resources– when Push comes, send request to get newest data to
achieve realtime
87
Push - Push off
client server
88
Push - Push off
client server
T0T1
T2
89
Push - Push off
client server
T0T1
T2
T4T5
T6
90
Push - Push off
client server
T0T1
T2
T4T5
T6
T8T9
T10
91
Push - Push on
client server SNS
92
Push - Push on
client server
T0T1
T2
SNS
93
Push - Push on
client server
T0T1
T2
T22T23
T24
SNS
94
Push - Push on
client server
T0T1
T2
T22T23
T24
SNS
T30
95
Push - Push on
client server
T0T1
T2
T22T23
T24
T31T32
T33
SNS
T30
96
Fallback
● Don't polling everything● Every GET APIs provide UI operation
97
Fallback
98
RESTful API design
99
RESTful API Design
● HTTP method● Common examples● Is Graph API RESTful?● Actual example● Internal & Open API
100
GET - Retrieve a resource
● A request to the server should never change the resource state
101
GET - Retrieve a resource
● A request to the server should never change the resource state
● Incidental side effects are OK– Like logging
102
GET - Retrieve a resource
● A request to the server should never change the resource state
● Incidental side effects are OK– Like logging
● Common response code– 200 OK
103
GET - Retrieve a resource
● A request to the server should never change the resource state
● Incidental side effects are OK– Like logging
● Common response code– 200 OK– 301 Moved Permanently
104
GET - Retrieve a resource
● A request to the server should never change the resource state
● Incidental side effects are OK– Like logging
● Common response code– 200 OK– 301 Moved Permanently– 404 Not Found (related to DELETE)– 410 Gone (related to DELETE)
105
DELETE - Remove a resource
● A request to the server should destroy the resource & never refer to it again
106
DELETE - Remove a resource
● A request to the server should destroy the resource & never refer to it again
● Common response code– 200 OK– 202 Accepted– 204 No Content
107
POST
108
POST
● POST-to-append
109
POST
● POST-to-append● Overloaded POST
110
POST-to-append
● A request to the server should create a new resource
111
POST-to-append
● A request to the server should create a new resource
● Common response code– 201 Created
112
POST-to-append
● A request to the server should create a new resource
● Common response code– 201 Created (plus "Location" header)
113
POST-to-append
● A request to the server should create a new resource
● Common response code– 201 Created (plus "Location" header)– 202 Accepted
114
Overloaded POST
● Providing a block of data, such as the result of submitting a form, to a data-handling process
115
Overloaded POST
● Providing a block of data, such as the result of submitting a form, to a data-handling process
● "Data-handling process" can be anything
116
Overloaded POST
● Providing a block of data, such as the result of submitting a form, to a data-handling process
● "Data-handling process" can be anything– POST /users/sort
117
Overloaded POST
● Providing a block of data, such as the result of submitting a form, to a data-handling process
● "Data-handling process" can be anything– POST /users/sort– POST /login
118
Overloaded POST
● Providing a block of data, such as the result of submitting a form, to a data-handling process
● "Data-handling process" can be anything– POST /users/sort– POST /login– ...etc.
119
PUT - Update a resource
● A request to the server should modify the resource
120
PUT - Update a resource
● A request to the server should modify the resource ● Common response code
– 200 OK– 204 No Content
121
Idempotent & Safe
122
Idempotent & Safe
● Idempotent– A HTTP method can be called many times without
different outcomes
123
Idempotent & Safe
● Idempotent– A HTTP method can be called many times without
different outcomes● Safe
– Do not modify resources
124
Idempotent & Safe
Idempotent Safe
GET yes yes
DELETE yes no
POST no no
PUT yes no
125
Common examples
126
GET /users
127
GET /users
● Retrieve all users' brief
128
GET /users
● Retrieve all users' brief{ "users": [ { "name": "kewang", "slogan": "Hello World" }, { "name": "Tony Stark", "slogan": "I'm Iron Man" }, { "name": "America captain", "slogan": "U.S.A." } ]}
129
GET /users/:user_id
130
GET /users/:user_id
● Retrieve a specific user information
131
GET /users/:user_id
● Retrieve a specific user information
{ "name": "kewang", "slogan": "Hello World", "birthday": "19831108", "sex": 0, "email": [ "[email protected]", "[email protected]" ], "company": "mitake"}
132
GET /users/:user_id/avatars
133
GET /users/:user_id/avatars
● Retrieve a specific user's all avatars' URL
134
GET /users/:user_id/avatars
● Retrieve a specific user's all avatars' URL{ "avatars": [ { "url": "http://www.s3.com/abc.png", "created": 1413385358 }, { "url": "http://www.s3.com/def.png", "created": 1401239876 }, { "url": "http://www.s3.com/ghi.png", "created": 1348303844 } ] }
135
GET /users/:user_id/avatar
136
GET /users/:user_id/avatar
● Retrieve a specific user's avatar URL
137
GET /users/:user_id/avatar
● Retrieve a specific user's newest avatar URL
138
GET /users/:user_id/avatar
● Retrieve a specific user's newest avatar URL
{ "url": "http://www.s3.com/abc.png"}
139
Is Graph API RESTful?Facebook powered
140
Graph API - Overview
Graph API - Overview
● Everything is a "node" has a unique ID
142
Graph API - Overview
● Everything is a "node" has a unique ID● Two nodes connect each other with a "edge"
143
Graph API - Overview
● Everything is a "node" has a unique ID● Two nodes connect each other with a "edge"● A node information contains some "fields"
144
Graph API - Are two people friends?
● GET /{user-a-id}/friends/{user-b-id}
145
Graph API - Does someone like a Page?
● GET /{user-id}/likes/{page-id}
146
Graph API - Publishing new Status Updates
● POST /{user-id}/feed● POST /{page-id}/feed
147
Graph API - Uploading Photos
● POST /{user-id}/photos● POST /{page-id}/photos● POST /{album-id}/photos
148
So, Graph API is RESTful?
150
An actual exampleKick a user from system
151
V1 - Kick a user from system
152
V1 - Kick a user from system
● Kick by myself
153
V1 - Kick a user from system
● Kick by myself– DELETE /me
154
V1 - Kick a user from system
● Kick by myself– DELETE /me
● Kick somebody
155
V1 - Kick a user from system
● Kick by myself– DELETE /me
● Kick somebody– DELETE /:user_id
156
V2 - Restore a user to system
157
V2 - Restore a user to system
● PUT /:user_id
158
V2 - Restore a user to system
● PUT /:user_id– Restore somebody to system
159
V2 - Restore a user to system
● PUT /:user_id– Restore somebody to system– Change somebody information
160
V3 - Update a user status
161
V3 - Update a user status
● PUT /:user_id/status
162
V3 - Update a user status
● PUT /:user_id/status– Kick somebody from system
163
V3 - Update a user status
● PUT /:user_id/status– Kick somebody from system– Restore somebody to system
164
Internal & Open API
165
Internal API design guideline
166
Internal API design guideline
● Minimize data size
167
Internal API design guideline
● Minimize data size● Merge operations
168
Internal API design guideline
● Minimize data size● Merge operations● Field names are short & convoluted
169
Internal API design guideline
● Minimize data size● Merge operations● Field names are short & convoluted● Customize request & response for official apps
170
Open API design guideline
171
Open API design guideline
● Minimize data size
172
Open API design guideline
● Minimize data size● Field names are simple & clear
173
Open API design guideline
● Minimize data size● Field names are simple & clear● Build request & response generally
174
Differences - GET /users
Internal API{
"us": [{
"id": "2d3f1a",
"nm": "Kewang",
"el": "[email protected]"
}, {
"id": "9a4f57",
"nm": "Hans",
"el": "[email protected]"
}]
}
Open API
{
"users": [{
"id": "2d3f1a",
"name": "Kewang"
}, {
"id": "9a4f57",
"name": "Hans"
}]
}
175
API Blueprint
176
API Blueprint
● Introduction● Aglio● API-Mock● Postman
177
Introduction
● Web API Language● Pure Markdown● Design for Humans● Understandable by Machines● Powerful Tooling● Easy Lifecycle
178
Hello World
179
Complex example
180
Aglio - API Blueprint renderer
181
Aglio - Complex example
182
API-Mock & Postman
183
Live DEMO
184
References
185
References
186
References
● Browser 與 Server 持續同步的作法介紹● Comet (programming)● Engine.IO: the realtime engine● Engine.IO Protocol● API Design Optimized for Mobile Platform● HTTP GET with request body● Does Google treat 404 and 410 status code differen
tly
187
188
