1. Amazon DynamoDB AWS Black Belt Tech Webinar 2014 ()

2. Agenda NoSQLRDB DynamoDB 3. Agenda NoSQLRDB DynamoDB 4. NoSQL Neo4j Couchbase DynamoDB MongoDB Riak HBase Cassandra 5. NoSQL vs RDB BASE / / ACID / / NoSQL RDB 6. BASE? 7. Basically Available, Soft-state, Eventually consistent Basically Available: Soft-state: Eventually consistent: NoSQLBASE 8. CAP? 9. CAP 3 (21) Consistency = Availability = Partition-tolerance = http://ja.wikipedia.org/wiki/CAP DynamoDBAP C 10. NoSQL 11. NoSQL NoSQL NoSQL NoSQL 12. DynamoDBNoSQL as a Service 13. Agenda NoSQLRDB DynamoDB 14. DynamoDB NoSQL as a Service SSD 15. DynamoDB Amazon.com RDBMS RDBMS HW RDBMS 16. DynamoDB RDBMSAmazon Dynamo (DynamoDB) Amazon Dynamo HW HW DB Amazon S3 17. DynamoDB AWS Amazon Dynamo 18. DynamoDB 19. SPOF 3AZ 20. ReadWrite Read : 1,000 Write : 100 Read : 500 Write : 1,000 DB 21. 22. DynamoDB Write AZAck Read Read Consistent Read ReadWrite Capacity Unit 23. DynamoDB Read/Write GB GB$0.285(2014/05@) http://aws.amazon.com/jp/dynamodb/pricing/ 24. DynamoDB $0.00742 :10 /1 $0.00742 : 50 /1 11KB11 14KB11 12 25. DynamoDB KVS Web DB ID 26. DynamoDB 1. KeyIndex 2. Read/Write Thats it, write your code! 27. DynamoDB API SDK HTTPAPI Database Client SideService Side Client application 28. API PutItem UpdateItem GetItem DeleteItem Query Scan BatchWriteItem BatchGetItem CreateTable DescribeTable UpdateTable DeleteTable 29. API PutItem UpdateItem GetItem DeleteItem Query Scan BatchWriteItem BatchGetItem CreateTable DescribeTable UpdateTable DeleteTable SDK 30. AWS SDKs and CLI SDKCLI Java Python PHP .NET Ruby nodeJS iOS Android Javascript in the Browser AWS CLI 31. SDK Perl Net::Amazon::DynamoDB Erlang wagerlabs/ddb https://github.com/wagerlabs/ddb Go goamz https://github.com/crowdmob/goamz 32. Agenda NoSQLRDB DynamoDB 33. Hash key DynamoDB(1/2) Table Item Attribute1 (Number) Attribute2 (String) Attribute3 (Binary) Hash key Item Attribute1 (Number) Attribute2 (String) Attribute3 (Binary) Hash key Item Attribute1 (Number) Attribute2 (String) Attribute3 (Binary) Hash key Table Item Attribute1 (Number) Hash key Item Attribute1 (Number) Attribute3 (Binary) Hash key Item Attribute1 (Number) Attribute2 (String) Hash key ItemHash key Attribute Attribute put 34. Hash keyRange key DynamoDB(2/2) Table Item Attribute1 (Number) Attribute2 (String) Attribute3 (Binary) Hash key Item Attribute1 (Number) Attribute2 (String) Attribute3 (Binary) Hash key Item Attribute1 (Number) Attribute2 (String) Attribute3 (Binary) Hash key Table Item Attribute1 (Number) Hash key Item Attribute1 (Number) Attribute3 (Binary) Hash key Item Attribute1 (Number) Hash key ItemHash keyRange keyAttribute Attribute put Range key Range key Range key Attribute2 (String) Attribute2 (String) Attribute2 (String) Range key Range key Range key 35. (1/2) Table Hash key Hash keyRange key Hash key Range key Hash key Hash + Range 36. (2/2) Item RDB 1Attribute Attributes RDB Hash key, Range keyAttributes Attributes Attributes String Number Binary Set of String Set of Number Set of Binary 37. A,D B,E C,F 123456789 Parition1 Partition2 ParitionN Range key Partition Hash key Partition Partition DynamoDB Hash keyRange key(1/2) 38. A,D B,E C,F 123456789 Parition1 Partition2 PartitionN Hash keyRange key(2/3) Partition DynamoDB Partition Partition Partition 39. A,D B,E C,F 123456789 Parition1 Partition2 ParitionN Hash keyRange key(3/3) Partition Hash key : x x/N x/N x/N 40. +1 Local Secondary Index Range key Hash key Global Secondary Index Hash Key 41. Local Secondary Indexes(LSI) QueryIndex 5 Hash key LSIQueryRange key ForumName=S3 && Subnect=aaa LSIKey Query(Attribute LSI) ForumName=S3 && Replies >= 10 ForumName=S3 && LastPostDateTime >=.. Forum 42. LSI Range keyAttribute / Local Secondary Indexes(LSI) Partition 43. Local Secondary Indexes(LSI) Local Secondary Indexes Secondary Index Query RepliesIndex Query ForumNameS3 Replies9 ForumName Replies 3 3 Query RepliesIndex 44. Local Secondary Indexes(LSI) Attribute Attribute Read Write 45. Local Secondary Indexes(LSI) LSI Table Index Read Write Table Index Table Index Read 1 read 2 read 2 write 46. Global Secondary Indexes(GSI) Hash keyAttribute Range keyLSIUserId QueryGameTitle Query GameTitle=Galaxy Invaders & TopScore >= 100 47. Global Secondary Indexes(GSI) GSI Hash keyAttribute / LSI GSIHash keyPartition Partition 48. Global Secondary Indexes(GSI) Attribute Attribute GSIAttribute Write 49. Global Secondary Indexes(GSI) GSI Table Index Read Write Table Index Read 50. Global Secondary Indexes(GSI) GSI Table Index Read Write Read 1 read Table Index 1 write 1 write 51. LSI/GSI LSI/GSI RDB 52. (1/3) GetItem Hash key PutItem 1 Update 1 Delete 1 Query Hash keyRange key BatchGet Scan 53. (2/3) Read GetItemQueryConsistent Read Read WriteRead Capacity Unit2 Conditional Write / // 54. (3/3) UpdateItemAttribute AttributeUpdateItemPutAddDelete3 PutAttribute AddAttributeNumber/Set DeleteAttribute Atomic Counter AddAtomic 55. Agenda NoSQLRDB DynamoDB 56. KVS Web DB ID 57. 1 KVS 58. KVS IDKVS UserIdItem UserId (Hash) Name Nicknames Mail Address Interests aed9d Bob [ Rob, Bobby ] foo@example.com some address [ Car, Motor Cycle] edfg12 Alice [ Allie ] a8eesd Carol [ Caroline ] f42aed Dan [ Daniel, Danny ] Users Table DynamoDBauto_incrementID 59. 2 KVS 60. KVS IDKVS ItemExpire PHP SDKDynamoDBSession Handler http://docs.aws.amazon.com/aws-sdk-php/guide/latest/ feature-dynamodb-session-handler.html SessionId (Hash) Expiry _a12fh 2014-02-01 00:00:00 _ee12a 2014-02-02 00:00:00 Sessions Table 61. 62. User (Hash) Timestamp (Range) Opponent Result Alice 2014-02-21 12:21:20 Bob Lost Alice 2014-02-21 12:42:01 Bob Won Alice 2014-02-24 09:48:00 Dan Won Alice 2014-02-25 16:21:11 Charlie Won Battle History User(Alice)Timestamp7 Charlie 02-25 16:21 Won! Your Battle History Dan 02-24 09:48 Won! Alice 02-21 12:42 Won! 63. 64. Home My Posts My Profile UserA Hello World! 10:18 UserB Hello World! 10:12 UserA Hello World! 10:11 UserA Hello World! 10:18 UserA Hello World! 10:11 UserA Hello World! 10:09 Name: UserA Mail: foo Profile: some texts 65. Users TableFriends Table " 2 66. User (Hash) Nicknames Bob [ Rob, Bobby ] Alice [ Allie ] Carol [ Caroline ] Dan [ Daniel, Danny ] Users Table Item Attribute (string, number, binary, set) Primary Key (Hash) 67. User (Hash) Nicknames Bob [ Rob, Bobby ] Alice [ Allie ] Carol [ Caroline ] Dan [ Daniel, Danny ] Friends Table User (Hash) Friend (Range) Bob Alice Alice Bob Alice Carol Alice Dan Users Table Hash + Range Primary Key 68. Friends Table Users Table User (Hash) Nicknames Bob [ Rob, Bobby ] Alice [ Allie ] Carol [ Caroline ] Dan [ Daniel, Danny ] User (Hash) Friend (Range) Bob Alice Alice Bob Alice Carol Alice Dan Alice 1. Query (Table = Friends, Hash = Alice, Range = *) 2. BatchGetItem(Bob, Carol, Dan) 69. Images Table User Hash) Image Range) Date Link Bob aed4c 2013-10-01 s3:// Bob cf2e2 2013-09-05 s3:// Bob f93bae 2013-10-08 s3:// Alice ca61a 2013-09-12 s3:// Bob Bob Query (Table=Images, Hash= Bob, Range=*) ? 70. Images Table User Image Date Link Bob aed4c 2013-10-01 s3:// Bob cf2e2 2013-09-05 s3:// Bob f93bae 2013-10-08 s3:// Alice ca61a 2013-09-12 s3:// User Date Image Bob 2013-09-05 cf2e2 Bob 2013-10-01 aed4c Bob 2013-10-08 f93bae Alice 2013-09-12 ca61a Table ByDate Local Secondary Index Local Secondary Index Date 71. ImageTags Table Image User aed4c Alice aed4c Bob f93bae Alice f93bae Bob Image f93baeAlice PutItem(Table = ImageTags, Hash = f93bae, Range = Alice) Bob ? Image f93bae Query(Table = ImageTags, Hash = f93bae, Range = *) 72. ImageTags Table UserImageRange Global Secondary Index User (Hash) Image (Range) Bob aed4c Bob f93bae Alice aed4c Alice f93bae ByUser Global Secondary Index Image (Hash) User (Range) aed4c Alice aed4c Bob f93bae Alice f93bae Bob Table Bob Alice 73. 74. " Process1 JobID (Hash) Progress (LSI) Created Time aed4c Done 2014-01-01 00:00:00 aed4c Done 2014-01-01 00:00:02 f93bae InProcess 2014-01-01 00:01:10 f93bae NotYet 2014-01-01 00:01:18 75. S3 S3 S3 EMR EMR 76. S3 S3 Jobs Table JobID (Hash) Created Time(Range) Process1 Process2 aed4c 2014-01-01 00:00:00 Done InProcess aed4c 2014-01-01 00:01:00 NoYet NotYet Process1 1. Scan (Table = jobs, Process1 = NotYet) Process1 1. Scan (Table = jobs, Process1 = NotYet) 77. S3 S3 Jobs Table JobID (Hash) Created Time(Range) Process1 Process2 aed4c 2014-01-01 00:00:00 Done InProcess aed4c 2014-01-01 00:01:00 NoYet NotYet Process1Lock UpdateItem (Table = jobs, HashKey=aed4c, Action=Put, Attributes=Process1:InProce ss) Process1Lock Update (Lock) UpdateItem (Table = jobs, HashKey=aed4c, Action=Put, Attributes=Process1:InProce ss) 78. Agenda NoSQLRDB DynamoDB 79. CloudWatch(1/2) 80. CloudWatch(2/2) CloudWatch Read Capacity Read Capacity Unit Throttled Read Requests Capacity OverRead Write Capacity Write Capacity Unit Throttled Write Requests Capacity OverWrite Get Latency Put Latency Query Latency Scan Latency User Errors 81. DynamoDBRDB 1.AWS Data PipelineCross Region Replication DynamoDB IncrementalAttribute AttributeIncremental 2.Amazon Elastic MapReduce S3DynamoDBAmazon Elastic MapReducehive 3. DynamoDB 82. Agenda NoSQLRDB DynamoDB 83. SCANQUERYPutItem 84. DynamoDB Local / DynamoDB CI JAR (Java7) API http://bit.ly/1d9fN5c 85. Transaction Library for DynamoDB AWS SDK for Java SDKDynamoDBTransaction Transaction Transaction http://bit.ly/16KbppP 86. Amazon EMRHive CREATE EXTERNAL TABLE Audience( AudienceId Int, ActionTimestamp string, Action string ) STORED BY 'org.apache.hadoop.hive.dynamodb.DynamoDBStorageHandler' TBLPROPERTIES ("dynamodb.table.name" = Audience2","dynamodb.column.mapping" = AudienceId:AudienceId, ActionTimestamp:Timestamp, Aciton:Action ); hiveExternal Table DynamoDB http://amazn.to/19goT17 87. hiveS3 EMRhiveDynamoDBS3 External Table DynamoDBExternal Table SelectS3External Table InsertINSERT OVERWRITE TABLE s3_as_external_table SELECT * FROM dynamodb_as_external_table; 88. COPYRedshift http://amazn.to/19goT17 Amazon Redshift COPY audience FROM dynamodb://Audience2 CREDENTIALS 'aws_access_key_id=;aws_secret_access_key=' READRATIO 50; 89. Agenda NoSQLRDB DynamoDB 90. Webinar AWS http://aws.amazon.com/jp/aws-jp-introduction/ 91. NoSQLRDB NoSQLDynamoDB