Storm×couchbase serverで作るリアルタイム解析基盤
Preview:
Citation preview
- 1. Copyright NTT Communications Corporation. All rights
reserved. Storm Couchbase Server NTT 2015611
- 2. Copyright NTT Communications Corporation. All rights
reserved. 2 n Couchbase ServerCouchbase Japan n n Couchbase
ServerStorm n Couchbase Server
- 3. Copyright NTT Communications Corporation. All rights
reserved. 3 { tetsuyam, NTT, , }, twitter@tetsuyam_twt, [, , ]
- 4. Copyright NTT Communications Corporation. All rights
reserved. Hadoop, Spark, BigQuery, Redshift Storm, Norikra,
Dataow
- 5. Copyright NTT Communications Corporation. All rights
reserved. Nathan Marz 3 Batch Layer Speed Layer Serving Layer
Lambda Architecture Raw Data Raw Data Raw Data Batch Layer Speed
Layer Serving Layer Lambda Architecture
- 6. Copyright NTT Communications Corporation. All rights
reserved. Nathan Marz 3 Batch Layer Speed Layer Serving Layer
Lambda Architecture Raw Data Raw Data Raw Data Batch Layer Speed
Layer Serving Layer Lambda Architecture
- 7. Copyright NTT Communications Corporation. All rights
reserved. PUSH 7 n NTTPUSH n 1010PUSH n TBSTBS 10 TBS 100DL
PUSH
- 8. Copyright NTT Communications Corporation. All rights
reserved. 8 n 10 n n
- 9. Copyright NTT Communications Corporation. All rights
reserved. on Cloud TBS Storm, CouchbaseServer NTTcloud RabbitMQ
Storm PUSH n
- 10. Copyright NTT Communications Corporation. All rights
reserved. userIDactionIDtimestampcontentsID
XXXX131397239563377YYYY1 XXXX251397240192325YYYY2
1433151060:cid|gender|state 1433151060:cid|gender
1433151060:cidTimestamp: PUSH
- 11. Copyright NTT Communications Corporation. All rights
reserved. Apache Storm 11 n n BackTypeTwitterOSS n SpoutBolt n
https://storm.apache.org/ Spout Tuple bolt Tuple Tuple
- 12. Copyright NTT Communications Corporation. All rights
reserved. Apache Storm 12 n SPOF n Nimbus ZooKeeper Supervisor
Master Node Cluster Coordinator Worker processes ZooKeeper
ZooKeeper Supervisor Supervisor Worker Process Worker Process
Worker Process Nimbus
- 13. Copyright NTT Communications Corporation. All rights
reserved. 13 Push ZK ZK ZK Nim SV SV SV SV ZoneA ZoneB
- 14. Copyright NTT Communications Corporation. All rights
reserved. 14 n n IO Wait
- 15. Copyright NTT Communications Corporation. All rights
reserved. 15
- 16. Copyright NTT Communications Corporation. All rights
reserved. 16 Y^Y^Y^Y^Y^Y^Y
- 17. Copyright NTT Communications Corporation. All rights
reserved. 17 NODE1 NODE2 NODE3 1 2 3 1 23 Client Library Cluster
Map APP
- 18. Copyright NTT Communications Corporation. All rights
reserved. 18 NODE1 NODE2 NODE3 1 2 3 1 23 Client Library Cluster
Map APP
- 19. Copyright NTT Communications Corporation. All rights
reserved. 19 NODE1 NODE2 NODE3 1 2 3 1 23 Client Library Cluster
Map APP 4 Write Error
- 20. Copyright NTT Communications Corporation. All rights
reserved. 20 NODE1 NODE2 NODE3 1 2 3 1 23 Client Library Cluster
Map APP 4 Write Error
- 21. Copyright NTT Communications Corporation. All rights
reserved. 21 NODE1 NODE2 NODE3 1 2 3 1 23 Client Library Cluster
Map APP 4 Write Error 30
- 22. Copyright NTT Communications Corporation. All rights
reserved. 22 NODE1 NODE2 NODE3 1 2 3 1 23 Client Library Cluster
Map APP 4 Write Error 30 30 30
- 23. Copyright NTT Communications Corporation. All rights
reserved. 23 NODE1 NODE2 NODE3 1 2 3 1 23 Client Library Cluster
Map APP 4 Write Error 30 XDCR
- 24. Copyright NTT Communications Corporation. All rights
reserved. 24 Client Library Cluster Map APP ClusterA ClusterB XDCR
NODE1 NODE NODE NODE1 NODE NODE
- 25. Copyright NTT Communications Corporation. All rights
reserved. 25 Client Library Cluster Map APP ClusterA ClusterB XDCR
NODE1 NODE NODE NODE1 NODE NODE
- 26. Copyright NTT Communications Corporation. All rights
reserved. 26 Client Library Cluster Map APP ClusterA ClusterB XDCR
NODE1 NODE NODE NODE1 NODE NODE Write Error
- 27. Copyright NTT Communications Corporation. All rights
reserved. 27 Client Library Cluster Map APP ClusterA ClusterB XDCR
NODE1 NODE NODE NODE1 NODE NODE
- 28. Copyright NTT Communications Corporation. All rights
reserved. 28 n n 30 n XDCR
- 29. Copyright NTT Communications Corporation. All rights
reserved. IO Wait 29 RAM
- 30. Copyright NTT Communications Corporation. All rights
reserved. IO Wait 30 RAM
- 31. Copyright NTT Communications Corporation. All rights
reserved. IO Wait 31 RAM
- 32. Copyright NTT Communications Corporation. All rights
reserved. IO Wait 32 RAM Disk write queue IO WaitCPU
- 33. Copyright NTT Communications Corporation. All rights
reserved. IO Wait 33 RAM Disk write queue IO WaitCPU SV
Y^Y^Y^Y^Y
- 34. Copyright NTT Communications Corporation. All rights
reserved. IO Wait 34 Disk write queue IO WaitCPU SV Y^Y^Y^Y^Y
Couchbase ServerRAM CouchbaseRAM (Couchbase
http://www.couchbase.com/jp/couchbase-server/features
- 35. Copyright NTT Communications Corporation. All rights
reserved. IO Wait 35 RAM
- 36. Copyright NTT Communications Corporation. All rights
reserved. IO Wait 36 RAM tmpfs
- 37. Copyright NTT Communications Corporation. All rights
reserved. IO Wait 37 tmpfsUnixOStmpfs HDD (wikipedia
- 38. Copyright NTT Communications Corporation. All rights
reserved. IO Wait 38 tmpfsUnixOStmpfs HDD (wikipedia
- 39. Copyright NTT Communications Corporation. All rights
reserved. IO Wait 39 tmpfsUnixOStmpfs HDD (wikipedia
- 40. Copyright NTT Communications Corporation. All rights
reserved. IO Wait 40 Couchbase Server Couchbase 2 3 (Couchbase
Server http://www.couchbase.com/jp/couchbase-
server/architecture
- 41. Copyright NTT Communications Corporation. All rights
reserved. IO Wait 41 n Couchbase Server DB n n n tmpfs, SSD
- 42. Copyright NTT Communications Corporation. All rights
reserved. 42 n n Couchbase Server 1 n Couchbase ServerDB
- 43. Copyright NTT Communications Corporation. All rights
reserved. Couchbase Server 43 n DL n SV n n KVS JSON
- 44. Copyright NTT Communications Corporation. All rights
reserved. Couchbase ServerL 44 n Redis TTL n memcached Couchbase
Server Memcached
- 45. Copyright NTT Communications Corporation. All rights
reserved. 45