64
© 2013 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc. AWS マイスターシリーズ Amazon ElastiCache 2013.10.30 アマゾンデータサービスジャパン株式会社 ソリューションアーキテクト 松尾 康博 ソリューションアーキテクト 逸峰 re:Generate

[AWSマイスターシリーズ] Amazon ElastiCache

Embed Size (px)

Citation preview

  • 1. re:G eneAWS rateAmazon ElastiCache 2013.10.30 2013 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc.

2. ! ! ! ! ! ! Amazon ElastiCache ElastiCache for Memcached ElastiCache for Redis 2 2013 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc. 3. ! DB ! (DB) () APP)3 2013 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc. 4. Web+DB ! AppRDBMS1. 2. AppDB 3. DB 4. App4 2013 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc. 5. Web+DB ! AppRDBMSAppRDBMS5. App,DB 6. DB 5 2013 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc. 6. Web+DB ! DB DB AppRDBMSApp6 2013 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc. 7. Web+DB ! DB AppRDBMS 7 2013 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc. 8. Web+DB ! Insert / Update AppRDBMS 8 2013 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc. 9. Web() ! App AppLBApp 9 2013 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc. 10. ! ! ! ! ! ! Amazon ElastiCache ElastiCache for Memcached ElastiCache for Redis 10 2013 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc. 11. AWS & SDKs Java, PHP, .NET, Python, Ruby, node.jsIDE Eclipse Visual StudioAmazon SNS Amazon SQS Amazon SESAmazon CloudFrontAmazon EC2 Auto ScaleWeb Management ConsoleAWS IAM Amazon CloudWatch Amazon Cloud SearchAWS Elastic Beanstalk AWS Cloud Formation AWS OpsWorksDevelopment & Administration Amazon SWFElastic MapReduceAmazon Elastic TranscoderAmazon S3Application ServiceAmazon EBS Amazon Glacier AWS Storage GatewayAmazon RDS Amazon DynamoDB Amazon ElastiCache Amazon RedshiftInfrastructure Service & Amazon VPC / Amazon Elastic Load Balancer / Amazon Route 53 /AWS Direct ConnectRegion11AZAWS Geographical Regions, Availability Zones, Points of Presence 2013 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc. 12. AWS Amazon RDS NoSQLAmazon DynamoDB Amazon Redshift Amazon ElastiCache12 2013 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc. 13. Web() CloudFront ( )Amazon Route 53 () ELB () AZ-1cAZ-1aEC2EC2S3 RDSRDS 2013 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc. 14. Amazon ElastiCache ! EC2! 2(memcached, redis) ! ! VPC 2013 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc. 15. Memcached/Redis on EC2 ! OS memecached/ Redis Management Console/CLI/API! Firewall! AWS 2013 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc. 16. 128Hi-Mem 2XL 33.4 GB 13 EC2 Compute Units 4 virtual cores64 Memory (GB)Hi-Mem 4XL 68 GB 26 EC2 Compute Units 8 virtual cores321684Medium 3.35 GB 2 EC2 Compute Unit 1 virtual core1162124816Hi-Compute 6.6 GB 20 EC2 Compute Units 8 virtual coresLarge 7.1 GB 4 EC2 Compute Units 2 virtual coresMicro 213 MB Up to 2 ECUs (for short bursts)Enhanced XL 14.6 GB 13 EC2 Compute Units 4 virtual coresExtra Large 14.6 GB 8 EC2 Compute Units 4 virtual cores Small 1.3 GB 1 EC2 Compute Unit 1 virtual coreEnhanced 2XL 29.6 GB 26 EC2 Compute Units 8 virtual coresHi-Mem XL 16.6 GB 6.5 EC2 Compute Units 2 virtual cores 3264128EC2 Compute Units (ECU)http://aws.amazon.com/jp/elasticache/features-and-benefits/ 2013 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc. 17. Cache Node Typememcached(max_cache_memory)redis(maxmemory)cache.t1.micro213 MB136 MBcache.m1.small1.3 GB900 MB3.35 GB2.95 GB7.1 GB6.7 GBcache.m1.xlarge14.6 GB14.2 GBcache.m3.xlarge14.6 GBcache.m3.2xlarge29.6 GBcache.m2.xlarge16.7 GB16.3 GBcache.m2.2xlarge33.8 GB33.4 GBcache.m2.4xlarge68 GB67.6 GB6.6 GB6.2 GBcache.m1.medium cache.m1.largecache.c1.xlarge memcached http://aws.amazon.com/jp/elasticache/features-and-benefits/ 2013 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc. 18. CloudWatch ! 60 ( CPUUtilization, etc. ) (Evict, CurrConnections, etc.) http://docs.aws.amazon.com/AmazonElastiCache/latest/UserGuide/CacheMetrics.html 2013 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc. 19. (Cache Event) ! ElastiCache10SNSPush etc! QueueCache EventMessageAmazon SNSElastiCache Cache NodeAlarmEmail NotificationHTTP Notificationhttp://docs.aws.amazon.com/AmazonElastiCache/latest/UserGuide/ElastiCacheSNS.html 2013 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc. 20. VPC ! VPC Cache Subnet Group Default VPC! Security Group Public IP(IP) CIDR http://docs.aws.amazon.com/AmazonElastiCache/latest/UserGuide/ElastiCacheAndVPC.html 2013 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc. 21. ! Security Group EC2 EC2 Security Group ! ElastiCache VPC Default Cache Security Group() Cache Security GroupSecurity GroupEC2 ClassicCache Security GroupEC2 VPCVPC Security GroupVPC DefaultVPC Security Group http://docs.aws.amazon.com/AmazonElastiCache/latest/UserGuide/ CacheSecurityGroup.html 2013 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc. 22. ! ElastiCache ! Reboot ! TIPS DB Cache Event http://docs.aws.amazon.com/AmazonElastiCache/latest/UserGuide/VersionManagement.htmll http://docs.aws.amazon.com/AmazonElastiCache/latest/UserGuide/ ManagingCacheClusters.html#ManagingCacheClusters.MaintWindow in part without the express consent of Amazon.com, Inc. 2013 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or 23. ! ! ! memcached: max_cache_memory, num_threads redis: maxmemory http://docs.aws.amazon.com/AmazonElastiCache/latest/UserGuide/ CacheParameterGroups.html 2013 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc. 24. ElastiCache ! ($/) ! ElastiCache ElastiCacheEC2 AZ EC2 : AZEC2 $0.01/GB $0.01/GB Availability Zone - a$0AppAvailability Zone - bhttp://aws.amazon.com/jp/elasticache/pricing/ 2013 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc. 25. ! EC2! (70%) memcached, redis ( cache.m1.xlarge) ( memcached or redis) () (: ) ( or 3) http://aws.amazon.com/jp/elasticache/pricing/ http://aws.amazon.com/jp/elasticache/reserved-cache-nodes/ 2013 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc. 26. Simple Monthly Calculator ! Web http://calculator.s3.amazonaws.com/calc5.html?lng=ja_JP 2013 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc. 27. AWS ! AWS http://aws.amazon.com/jp/free/! ElastiCache http://aws.amazon.com/jp/elasticache/free/ cache.t1.micro 750/ 2013 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc. 28. ! ! ! ! ! ! Amazon ElastiCache ElastiCache for Memcached ElastiCache for Redis 28 2013 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc. 29. Memcached ! key-value 2003Danga Interactive(BSD) Live Journal (YouTube, Wikipedia, mixi, etc. )! KVS Key-value Telnet LRU3 2013 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc. 30. Amazon ElastiCache for memcached ! 1.4.51.4.14 (2013.10.30) memcached Cache ClusterCache Node! Cache Cluster Cache Cluster CacheCluster AAvailability Zone - aCacheCluster BAvailability Zone - b30 2013 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc. 31. Auto Discovery for memcached ! Cache Cluster! Auto Discovery(Java, PHP) Cache ClusterConfiguration Endpoint ! Configuration EndpointCache Cluster ( Proxy) Endpoint CacheClusterApp xxx.0001.apne1.cache.amazonaws.comApp Auto Discovery xxx.0002.apne1.cache.amazonaws.com Configuration Endpoint xxx.cfg.apne1.cache.amazonaws.comhttp://docs.aws.amazon.com/AmazonElastiCache/latest/UserGuide/AutoDiscovery.html 2013 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc. 32. CloudWatchmemcached ! http://docs.aws.amazon.com/AmazonElastiCache/latest/ UserGuide/CacheMetrics.Memcached.html! CPUUtilization (CPU) Evictions ( ) SwapUsage CurrConnections http://docs.aws.amazon.com/AmazonElastiCache/latest/ UserGuide/CacheMetrics.WhichShouldIMonitor.html 2013 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc. 33. ! ! ! ! ! ! Amazon ElastiCache ElastiCache for Memcached ElastiCache for Redis 33 2013 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc. 34. Redis ! In-memory Key-Value Store ! List, Set, Sorted Set, Hash! Snapshot, Append only File! Replication! Pub/Sub ! Lua scripting http://redis.io/ 34 2013 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc. 35. Key-ValueValue Value35http://redis.io/commands 2013 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc. 36. Pub/Sub / subscribe delivermsgClient 1Client 2 msgpublishe subscribeChannels / Patterns36deliverClient 3 msghttp://redis.io/topics/pubsub 2013 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc. 37. ElastiCache for Redis ! : 2.6.13 (2013.10.30) Replication S3(RDB) Redis Lua Scripting Pub/Sub Append Only File! CONFIG, SLAVEOF () 2013 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc. 38. (Replication) ! () Read! Replication Group 5 Replica of Replica Replication Group CacheClusterCacheClusterAvailability Zone - aCacheClusterAvailability Zone - bhttp://docs.aws.amazon.com/AmazonElastiCache/latest/UserGuide/Replication.html http://docs.aws.amazon.com/AmazonElastiCache/latest/UserGuide/ManagingReplication.html 2013 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc. 39. Replication AZ AZAppGETSETSETAvailability Zone - aAppGETAvailability Zone - b 2013 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc. 40. endpoint unlink SYNC DNS change 2013 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc. 41. 1. SNS 1. SNS 2. 2. 3. FO 3. DNS 4. FODNS FO FORR SNS / retry, DB SNS http://aws.amazon.com/jp/elasticache/faqs/#rf7 http://aws.amazon.com/jp/elasticache/faqs/#rf8 2013 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc. 42. Sentinel vs. ElastiCache(Redis) Redis Sentinel RedisHA SentinelElastiCache(endpoint )(endpoint ) http://redis.io/topics/sentinel 2013 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc. 43. RDB ! Redis RedisRDBS3 S3RDB! RDB S3ElastiCache RDB ()Redis EC2RDBcopyS3ElastiCache forRedishttp://docs.aws.amazon.com/AmazonElastiCache/latest/UserGuide/ ManagingCacheClusters.html#ManagingCacheClusters.SeedingRedis 2013 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc. 44. S3 S3(: mybacket/path/data.rdb) 2013 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc. 45. : EC2Redisslave EC2Redis slaveRDBS3 S3slaveofRedisrestoreRDBcopyEC2S3http://redis.io/topics/persistence http://docs.aws.amazon.com/AmazonElastiCache/latest/UserGuide/ ManagingCacheClusters.html#ManagingCacheClusters.RedisSnapshots 2013 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc. 46. Append-Only Files(AOF) ! AOF ()AOF reboot reboot! off appendonly yes cache.t1.micro( appendonly=yes)! AOF http://docs.aws.amazon.com/AmazonElastiCache/latest/UserGuide/ CacheParameterGroups.Redis.html#CacheParameterGroups.Redis.AOF 2013 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc. 47. CloudWatchRedis ! CPUUtilization (CPU) Evictions ( ) CurrConnections Replica Lag ( )! CPU Redis 1 cache.m1.xlarge(4) 25% (100% / 4) Alert http://docs.aws.amazon.com/AmazonElastiCache/latest/ UserGuide/CacheMetrics.WhichShouldIMonitor.html http://docs.aws.amazon.com/AmazonElastiCache/latest/UserGuide/CacheMetrics.html 2013 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc. 48. ! ! ! ! ! ! Amazon ElastiCache ElastiCache for Memcached ElastiCache for Redis 48 2013 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc. 49. ! memcached , redis ! redis (atomic counter) (bitmaps) (lists) (pub/sub) , (sorted sets)49 2013 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc. 50. (broadcastwithRedispub/sub)*Image from: https://geoloqi.com/blog/2011/09/building-a-real-time-location-based-urban-geofencing-game-with-socket-io-redisnode-js-and-sinatra-synchrony/ 2013 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc. 51. ! Web9:1 DB! ! DB 51 2013 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc. 52. ! DB ()AppRDBMS App 2013 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc. 53. (Consistent Hashing) ! Consistent Hashing AppConsistent Hashing APP CH 2013 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc. 54. ! PHPlibmemcacheddistribution !Twemproxy TwitterMemcached/Redisproxy (consistent hashing) ! Consistent hashing - Wikipedia, the free encyclopedia http://en.wikipedia.org/wiki/Consistent_hashing memcached4memcached gihyo.jp http://gihyo.jp/dev/feature/01/memcached/0004?page=3 Partitioning: how to split data among multiple Redis instances. http://redis.io/topics/partitioning 2013 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc. 55. Twemproxy TwemproxySPOF Twemproxy twemproxyRedismemcached App + TwemproxyApp + TwemproxyApp + TwemproxyRG1App + TwemproxyRG255 2013 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc. 56. ! ! ! ! ! ! Amazon ElastiCache ElastiCache for Memcached ElastiCache for Redis 56 2013 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc. 57. Amazon ElastiCache ! Management Console API (redis)! VPC57 2013 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc. 58. ! Amazon ElastiCache http://aws.amazon.com/jp/elasticache/! http://aws.amazon.com/jp/elasticache/pricing/ http://aws.amazon.com/jp/elasticache/reserved-cache-nodes/! FAQ http://aws.amazon.com/jp/elasticache/faqs/! http://aws.amazon.com/jp/documentation/elasticache/! Amazon ElastiCache https://forums.aws.amazon.com/forum.jspa?forumID=127 2013 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc. 59. APPENDIX 2013 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc. 60. Appendix: Redis ! (unknown command) (config, slaveof) ( save, bgsave, bgrewriteof, migrate) ( shutdown) debug! http://redis.io/commands http://docs.aws.amazon.com/AmazonElastiCache/latest/ UserGuide/ClientConfig.html#ClientConfig.RestrictedCommands 2013 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc. 61. Appendix: Redis ! activerehashing=Yes databases=16 lua-time-limit maxclients maxmemory ( maxmemory-policy)! http://redis.io/topics/config http://docs.aws.amazon.com/AmazonElastiCache/latest/ UserGuide/CacheParameterGroups.Redis.html 2013 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc. 62. Appendix: Scale-up vs. Scale-out Scale-upScale-out Library/proxy RedisFO (68GB) 2013 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc. 63. Appendix: GB m2 0.16 0.14Case Study 4GB m1.large 32GB m2.2xlarge (scale-up) m2.xlarge X 2 (Hybrid) m1.large X 6 (scale-out)0.12 0.1 0.08 0.06$/h/GB0.04 0.02 0*Y: 1GB *memcached max_cache_memory Redis 2013 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc. 64. Appendix: m1.medium $0.206 Cost / Performance t1.micro m1.small m1.medium m1.large m2.xlargeCost/Performance 1.2 1 0.8 0.6 0.4 0.2CP0*Y: m2.4xlarge1 2013 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified or distributed in whole or in part without the express consent of Amazon.com, Inc.