Case Study on AWS JavaScript SDK and DynamoDB

Preview:

Click to see full reader

DESCRIPTION

Case study on how to manipulate AWS DynamoDB as well as IAM / STS with their JavaScript SDK in the Browser. I keep notes in memo and comments a lot, so please download and read it if you're really interested. It's a powerpoint, and if that's a problem, please let me know. I'll try convert it to PDF or some other open / free formats. Licensed under CC-BY / MIT (demo project).

Citation preview

AWS JavaScript SDK

and DynamoDBCliff Chao-kuan Lu

<clifflu@gmail.com>

A Case Study:

May 21st ‘14, AWS User Group Taiwan

04/07/2023CC-BY 3.0, Cliff Chao-kuan Lu <clifflu@gmail.com>

2

授權與格式說明 以超連結 (hyperlink) 代替引用 (attribution)

引用外部內容均◦已取得授權，或◦包含原始連結，並在合理範圍內引用

本文件原創內容以 CC-BY 3.0 釋出

https://creativecommons.org/licenses/by/3.0/tw/

04/07/2023CC-BY 3.0, Cliff Chao-kuan Lu <clifflu@gmail.com>

3

About cliffluFull-stack Web DeveloperAWS Solutions ArchitectNerd

about.me/clifflu

http://about.me/clifflu
http://about.me/clifflu

04/07/2023CC-BY 3.0, Cliff Chao-kuan Lu <clifflu@gmail.com>

4

About cliffluFull-stack Web DeveloperAWS Solutions ArchitectNerd

about.me/clifflu

Level Up !! Professional

http://about.me/clifflu
http://about.me/clifflu

04/07/2023CC-BY 3.0, Cliff Chao-kuan Lu <clifflu@gmail.com>

5

Intro◦AWS JS SDK◦IAM◦DynamoDB

Case Study◦Headless Poller

大綱

04/07/2023CC-BY 3.0, Cliff Chao-kuan Lu <clifflu@gmail.com>

6

Everything Changes

04/07/2023CC-BY 3.0, Cliff Chao-kuan Lu <clifflu@gmail.com>

7

AWS SDK for JavaScript in the Browser

這名字好長

Part 1.1

04/07/2023CC-BY 3.0, Cliff Chao-kuan Lu <clifflu@gmail.com>

8

很新AWS JS SDK / Browser

http://aws.typepad.com/aws/2013/10/developer-preview-aws-sdk-for-javascript.html?utm_source=feedburner&utm_medium=feed&utm_campaign=Feed:+AmazonWebServicesBlog+(Amazon+Web+Services+Blog)

04/07/2023CC-BY 3.0, Cliff Chao-kuan Lu <clifflu@gmail.com>

9

官網 起自 AWS SDK for Node.js2.0 更名並支援 Browser

AWS JS SDK / Browser

http://aws.amazon.com/developers/getting-started/browser/

04/07/2023CC-BY 3.0, Cliff Chao-kuan Lu <clifflu@gmail.com>

10

率先支援 ◦DynamoDB◦S3◦SNS◦SQS

猛烈 rc 中

服務

https://aws.amazon.com/releasenotes/JavaScript?c=25&p=1&sm=dD

04/07/2023CC-BY 3.0, Cliff Chao-kuan Lu <clifflu@gmail.com>

11

AutoScaling, CloudFormation, CloudFront, Cloudsearch, CloudTrail, CloudWatch, DataPipeline, DirectConnect, DynamoDB, EC2, ElastiCache, ElasticBeanstalk, ElasticTranscoder, ELB, EMR, Glacier, IAM, ImportExport, Kinesis, OpsWorks, RDS, Redshift, Route53, S3, SES, SimpleDB, SNS, SQS, StorageGateway, STS, Support, SWF

服務

04/07/2023CC-BY 3.0, Cliff Chao-kuan Lu <clifflu@gmail.com>

12

支援

http://docs.aws.amazon.com/AWSJavaScriptSDK/guide/browser-intro.html
http://docs.aws.amazon.com/AWSJavaScriptSDK/guide/browser-intro.html
http://docs.aws.amazon.com/AWSJavaScriptSDK/guide/browser-intro.html

04/07/2023CC-BY 3.0, Cliff Chao-kuan Lu <clifflu@gmail.com>

13

IE…

04/07/2023CC-BY 3.0, Cliff Chao-kuan Lu <clifflu@gmail.com>

14

用例國防布

04/07/2023CC-BY 3.0, Cliff Chao-kuan Lu <clifflu@gmail.com>

15

SDK for◦PHP, Python, Node.js◦Java, .NET, ◦Ruby

AWS SDK for Android & iOSDec. 8th, 2010

親友

04/07/2023CC-BY 3.0, Cliff Chao-kuan Lu <clifflu@gmail.com>

16

多此一舉？

04/07/2023CC-BY 3.0, Cliff Chao-kuan Lu <clifflu@gmail.com>

17

Browser 直通 AWS◦減低對 API Server 的需求

根本

04/07/2023CC-BY 3.0, Cliff Chao-kuan Lu <clifflu@gmail.com>

18

機房、機器、網路、電力很難搞交給 Amazon Web Services 正好 那 EC2 Instance 呢？

反思

04/07/2023CC-BY 3.0, Cliff Chao-kuan Lu <clifflu@gmail.com>

19

EC2 各種麻煩◦Load Balancing◦Types◦Contracts◦AutoScaling Parameters Pattern

淵藪

04/07/2023CC-BY 3.0, Cliff Chao-kuan Lu <clifflu@gmail.com>

20

太多 Gotcha 每層服務、各層之間都要考慮 複雜度可能變 M x N

Scaling & HA

04/07/2023CC-BY 3.0, Cliff Chao-kuan Lu <clifflu@gmail.com>

21

範例example.com

ELBRoute53

CF

S3StaticConte

nt

SharedEnv

Auto Scaling group

AMI

AZ 2Web

ServersS1

Secondary

S2 Secondary

Config

AZ 1Web Servers

S1 Primary

S2 Primary

Config

AZ 3Web Servers

Config + Arbitor

mongod

04/07/2023CC-BY 3.0, Cliff Chao-kuan Lu <clifflu@gmail.com>

22

將麻煩留給 AWS◦第三方與服務端授權◦Scaling / HA

優點

04/07/2023CC-BY 3.0, Cliff Chao-kuan Lu <clifflu@gmail.com>

23

促成符合 SOA Pattern 之架構優點Web Page

HTML

CSSJS

Authenticate &

Authorize

Services

1. Auth Request

3. Authorized Identity

2. Access Token

04/07/2023CC-BY 3.0, Cliff Chao-kuan Lu <clifflu@gmail.com>

24

容易整合其他服務 IAM

STS: Security Token Service WIF: Web Identity Federation

DynamoDB, S3, … 自有服務， SOA 嘛 O.o/

優點

04/07/2023CC-BY 3.0, Cliff Chao-kuan Lu <clifflu@gmail.com>

25

減低將 Access / Secret Token 打入源碼的可能性優點

04/07/2023CC-BY 3.0, Cliff Chao-kuan Lu <clifflu@gmail.com>

26

API IAMServices

基石

04/07/2023CC-BY 3.0, Cliff Chao-kuan Lu <clifflu@gmail.com>

27

RESTful, SOAP (deprecated)Dev Tools 是好朋友API

04/07/2023CC-BY 3.0, Cliff Chao-kuan Lu <clifflu@gmail.com>

28

安全乃第一要務 不然會變成礦工 ˇˇ然後

04/07/2023CC-BY 3.0, Cliff Chao-kuan Lu <clifflu@gmail.com>

29

Identity and Access Management

AWS 權限樞紐

Part 1.2

04/07/2023CC-BY 3.0, Cliff Chao-kuan Lu <clifflu@gmail.com>

30

Root UserGroup / IAM UserRoles

◦AWS SVC◦X-Account◦IdP Web SAML

IAM: Identity

04/07/2023CC-BY 3.0, Cliff Chao-kuan Lu <clifflu@gmail.com>

31

Password MFA (Multi-factor authentication)

Access / Secret Key Pair X.509 certificate

3rd Party ◦SAML◦Web Identity Federation

IAM: Authentication

04/07/2023CC-BY 3.0, Cliff Chao-kuan Lu <clifflu@gmail.com>

32

IAM: Policy

04/07/2023CC-BY 3.0, Cliff Chao-kuan Lu <clifflu@gmail.com>

33

IAM: Policy Effect: Deny | Allow Action:

◦允許呼叫的 API Resource:

◦arn Principal

◦授權端限制 Condition

◦其他限制

04/07/2023CC-BY 3.0, Cliff Chao-kuan Lu <clifflu@gmail.com>

34

for rule in rules:◦Explicit Deny -> Deny◦Explicit Allow -> Allow◦Default Deny

稽核

04/07/2023CC-BY 3.0, Cliff Chao-kuan Lu <clifflu@gmail.com>

35

IAM: arn

格式◦冒號分隔◦首二節固定為 arn:aws◦Service◦Region◦Account◦Resource Identifier

04/07/2023CC-BY 3.0, Cliff Chao-kuan Lu <clifflu@gmail.com>

36

給路人甲的◦Access / Secret Key Pair◦Management Console 登入權限

IAM : STS

04/07/2023CC-BY 3.0, Cliff Chao-kuan Lu <clifflu@gmail.com>

37

第三方驗證，歡迎大家路過不用怕被關

04/07/2023CC-BY 3.0, Cliff Chao-kuan Lu <clifflu@gmail.com>

38

Web Identity Federation

Facebook, Google 可也

http://docs.aws.amazon.com/STS/latest/UsingSTS/STSUseCases.html

04/07/2023CC-BY 3.0, Cliff Chao-kuan Lu <clifflu@gmail.com>

39

只接受下列驗證機制◦表三家： Amazon, Facebook, Google◦SAML 說明列表 IAM Partners

限制

http://docs.aws.amazon.com/IAM/latest/UserGuide/IdP-solution-providers.html
http://aws.amazon.com/iam/partners/

04/07/2023CC-BY 3.0, Cliff Chao-kuan Lu <clifflu@gmail.com>

40

Trust Relationships◦Identity Provider◦Client ID

Permissions

IAM Role for WIF

04/07/2023CC-BY 3.0, Cliff Chao-kuan Lu <clifflu@gmail.com>

41

用戶可透過第三方驗證與 IAM:STS, WIF 授予調用 AWS API 之權限

至此

04/07/2023CC-BY 3.0, Cliff Chao-kuan Lu <clifflu@gmail.com>

42

WIF PlaygroundLogin with amazonAWS Documentation

◦Using IAM◦Using STS◦SDK for JavaScript

參照

https://web-identity-federation-playground.s3.amazonaws.com/index.html
http://login.amazon.com/website

04/07/2023CC-BY 3.0, Cliff Chao-kuan Lu <clifflu@gmail.com>

43

DynamoDBManaged NoSQL Service

Part 1.3

04/07/2023CC-BY 3.0, Cliff Chao-kuan Lu <clifflu@gmail.com>

44

Managed NoSQL Service 取代 SimpleDB 三本柱

◦Scalable◦Available◦Fast

DynamoDB

04/07/2023CC-BY 3.0, Cliff Chao-kuan Lu <clifflu@gmail.com>

45

Data ModelScalar

◦Number{“N”: “300”}

◦String{“S”: “300”}

◦Binary{“B”: “BASE64”}

Multi-valued◦Number Set

{“NS”: [“1”,”2”,”3.14”]}◦String Set

{“SS”: [“A”,”b”]}◦Binary Set

{“BS”: [“BASE64”]}

04/07/2023CC-BY 3.0, Cliff Chao-kuan Lu <clifflu@gmail.com>

46

Document size: 64 KB (UTF-8) 5 LSIs / 5 GSIs per Table Min throughput: 1

Hash Key: 2 KB Range Key: 1KB

BatchGetItem: 1MB or 100 items BatchWriteItem: 1MB or 25 items

Limits

04/07/2023CC-BY 3.0, Cliff Chao-kuan Lu <clifflu@gmail.com>

47

Data Storage:◦約是 S3 10x

Provisioned Throughputs◦Reads ： 4kb 循序可合併 Eventually Consistent 消耗減半

◦Writes : 1kb

Pricing

04/07/2023CC-BY 3.0, Cliff Chao-kuan Lu <clifflu@gmail.com>

48

Hash Key [opt] Range Key

Primary Key

hash: “a”

hash: “b”

{“hash”: “a”, “range”: “123”, …}{“hash”: “a”, “range”: “223”, …}{“hash”: “a”, “range”: “321”, …}

{“hash”: “b”, “range”: “3”, …}{“hash”: “b”, “range”: “22”, …}{“hash”: “b”, “range”: “321”, …}

04/07/2023CC-BY 3.0, Cliff Chao-kuan Lu <clifflu@gmail.com>

49

Primary Key

hash: “a”

hash: “b”

{“hash”: “a”, “range”: “123”, …}{“hash”: “a”, “range”: “223”, …}{“hash”: “a”, “range”: “321”, …}

{“hash”: “b”, “range”: “3”, …}{“hash”: “b”, “range”: “22”, …}{“hash”: “b”, “range”: “321”, …}

SortedSharded

04/07/2023CC-BY 3.0, Cliff Chao-kuan Lu <clifflu@gmail.com>

50

Index Name

Hash KeyRange KeyProjection

Shared Throughputs

Local Secondary Index

04/07/2023CC-BY 3.0, Cliff Chao-kuan Lu <clifflu@gmail.com>

51

Index Name / Hash / Range Key ◦比照 LSI◦不要求 uniqueness

有自己的 Throughputs

Eventual ConsistencyProjected Attributes

Global Secondary Index

04/07/2023CC-BY 3.0, Cliff Chao-kuan Lu <clifflu@gmail.com>

52

針對特定 Index (PK, LSI or GSI) 查詢 支援 Condition 支援 Filter 高效

Query

04/07/2023CC-BY 3.0, Cliff Chao-kuan Lu <clifflu@gmail.com>

53

現適用於 scan 及 query 消耗 throughput 不變 減低 DynamoDB <-> Caller 傳輸boto 僅 boto.dynamodb2 支援

Filter

04/07/2023CC-BY 3.0, Cliff Chao-kuan Lu <clifflu@gmail.com>

54

…Scan

04/07/2023CC-BY 3.0, Cliff Chao-kuan Lu <clifflu@gmail.com>

55

Upsert 需包含 Primary Key

Update: PATCHPut: POST (PUT)

支援 Conditional Operation

Put, Update

04/07/2023CC-BY 3.0, Cliff Chao-kuan Lu <clifflu@gmail.com>

56

支援 Conditional Operation 用於刪除 Item

若要刪除 Attribute ，需使用 updateItem

Delete

04/07/2023CC-BY 3.0, Cliff Chao-kuan Lu <clifflu@gmail.com>

57

“Expected” 可實現 MVCC pattern

配合 Document Atomicity 可模擬transactional behavior (2-Phase Commit)

Conditional Operation

04/07/2023CC-BY 3.0, Cliff Chao-kuan Lu <clifflu@gmail.com>

58

updateItem◦AttributeUpdates Value Action:

PUT DELETE ADD

Atomic Increment

04/07/2023CC-BY 3.0, Cliff Chao-kuan Lu <clifflu@gmail.com>

59

Fine-Grained Policies

Item

Attribute

04/07/2023CC-BY 3.0, Cliff Chao-kuan Lu <clifflu@gmail.com>

60

Fine-Grained Policies

Item◦“dynamodb:LeadingKeys”: [“xxx”]

Attribute◦“dynamodb:Attributes”: [“xxx”,”yyy”]

04/07/2023CC-BY 3.0, Cliff Chao-kuan Lu <clifflu@gmail.com>

61

While True: Item.save(expect=…)

1unit for Read / Write

More on Throughput

04/07/2023CC-BY 3.0, Cliff Chao-kuan Lu <clifflu@gmail.com>

62

Stats from DynamoDB

More on Throughput

04/07/2023CC-BY 3.0, Cliff Chao-kuan Lu <clifflu@gmail.com>

63

CloudWatch 顯示低階數據`Expect` 不消耗 Read Unit 允許 short burst

猜想

04/07/2023CC-BY 3.0, Cliff Chao-kuan Lu <clifflu@gmail.com>

64

了解 DynamoDB 的◦設計目標◦調校 Index Throughput

◦操作◦計費

小結

04/07/2023CC-BY 3.0, Cliff Chao-kuan Lu <clifflu@gmail.com>

65

Headless-Poller終於到正題了 !?

Part 2

04/07/2023CC-BY 3.0, Cliff Chao-kuan Lu <clifflu@gmail.com>

66

Phil: clifflu 你要不要講五月小聚

Henry: 聽說 Cloudflare + wordpress 會爆炸clifflu: 好啊，就講這個吧

緣起

04/07/2023CC-BY 3.0, Cliff Chao-kuan Lu <clifflu@gmail.com>

67

秒殺WP 看 location.href 重導頁面 講不滿三十分鐘 ˇˇ 只好調出備用題目

然後

04/07/2023CC-BY 3.0, Cliff Chao-kuan Lu <clifflu@gmail.com>

68

EC2 各種麻煩◦Load Balancing◦Types◦Contracts◦AutoScaling Parameters Pattern

回想

04/07/2023CC-BY 3.0, Cliff Chao-kuan Lu <clifflu@gmail.com>

69

就來演示不靠 EC2 的小服務吧

04/07/2023CC-BY 3.0, Cliff Chao-kuan Lu <clifflu@gmail.com>

70

使用服務◦IAM, S3 (DynamoDB), CF

Octopress / Jekyll !?Clone Ruby 好像很遜 >///<

寫個 Blog system ?

04/07/2023CC-BY 3.0, Cliff Chao-kuan Lu <clifflu@gmail.com>

71

最新功能最冷門功能最好 Manual 沒有

必須霸氣外露！

04/07/2023CC-BY 3.0, Cliff Chao-kuan Lu <clifflu@gmail.com>

72

前端◦Angular.js 潮◦OAuth 勁

後端◦從缺，帥

資料◦DynamoDB ，猛

線上投票

04/07/2023CC-BY 3.0, Cliff Chao-kuan Lu <clifflu@gmail.com>

73

前端◦Angular.js 潮◦OAuth 勁

後端◦從缺，帥

資料◦DynamoDB ，猛

線上投票需要 https

CloudFront

04/07/2023CC-BY 3.0, Cliff Chao-kuan Lu <clifflu@gmail.com>

74

申請 APP◦Amazon◦Facebook◦Google

將 access_token 透過 STS 轉換為 Access / Secret Key Pair

第三方授權

04/07/2023CC-BY 3.0, Cliff Chao-kuan Lu <clifflu@gmail.com>

75

欄位： uid, q_id, o_id

Primary Key◦(uid, q_id) : unique 確保每人每題一票◦選 uid 為 hash key Authentication Cardinality

DynamoDB

04/07/2023CC-BY 3.0, Cliff Chao-kuan Lu <clifflu@gmail.com>

76

updateItem (省略 callback)投票

Primary Key

Upsert

Return on Update

04/07/2023CC-BY 3.0, Cliff Chao-kuan Lu <clifflu@gmail.com>

77

updateItem權限

Hash Key

04/07/2023CC-BY 3.0, Cliff Chao-kuan Lu <clifflu@gmail.com>

78

正解： Worker ，但霸氣不足 需求：

◦uid 不外漏◦affordable◦快

計票

04/07/2023CC-BY 3.0, Cliff Chao-kuan Lu <clifflu@gmail.com>

79

建立 (q_id, o_id) 之 GSI禁止讀取 uid 欄位 取出列數即為總票數 循序讀取，節省 read capacity

計票： GSI

04/07/2023CC-BY 3.0, Cliff Chao-kuan Lu <clifflu@gmail.com>

80

LSI / GSI 必定包含 Primary Key 透過 Query / Scan 取得 Item 時，必須允許讀取 primary key

麻煩，天大的麻煩

04/07/2023CC-BY 3.0, Cliff Chao-kuan Lu <clifflu@gmail.com>

81

曙光

04/07/2023CC-BY 3.0, Cliff Chao-kuan Lu <clifflu@gmail.com>

82

運用 “ select”: “COUNT” 只計算票數，不取 Item body

循序讀取

計票： GSI

04/07/2023CC-BY 3.0, Cliff Chao-kuan Lu <clifflu@gmail.com>

83

Document 沒寫Boto 沒使用

◦dynamodb 實做了 query.count() ，透過取回 item 記數◦dynamodb2 未有類似功能

霸氣

04/07/2023CC-BY 3.0, Cliff Chao-kuan Lu <clifflu@gmail.com>

84

GSI Read Capacity, 1.5 !!??穩定

04/07/2023CC-BY 3.0, Cliff Chao-kuan Lu <clifflu@gmail.com>

85

權限

04/07/2023CC-BY 3.0, Cliff Chao-kuan Lu <clifflu@gmail.com>

86

計票

04/07/2023CC-BY 3.0, Cliff Chao-kuan Lu <clifflu@gmail.com>

87

終於

04/07/2023CC-BY 3.0, Cliff Chao-kuan Lu <clifflu@gmail.com>

88

參數Throughpu

tsRead Write

Table 1 5GSI 19 5

04/07/2023CC-BY 3.0, Cliff Chao-kuan Lu <clifflu@gmail.com>

89

參數Throughpu

tsRead Write

Table 1 5GSI 19 5

Total #Reads8kb data 消耗 1

投票消耗GSI ~ Table

~6.6 USD/mo

04/07/2023CC-BY 3.0, Cliff Chao-kuan Lu <clifflu@gmail.com>

90

改用 Batched Query ?◦不支援 select COUNT

優化流程◦資料更新◦後台計票◦操作介面◦PR please

改進？

04/07/2023CC-BY 3.0, Cliff Chao-kuan Lu <clifflu@gmail.com>

91

講完了今天沒有 bonus session

04/07/2023CC-BY 3.0, Cliff Chao-kuan Lu <clifflu@gmail.com>

92

Q & A回家記得念 Manual

04/07/2023CC-BY 3.0, Cliff Chao-kuan Lu <clifflu@gmail.com>

93

/clifflu/headless-poller

簡報下載： AWS Doc

◦IAM◦STS◦DynamoDB

API

Links

https://github.com/clifflu/headless-poller
https://github.com/clifflu/headless-poller
http://docs.aws.amazon.com/IAM/latest/UserGuide/IAM_Introduction.html
http://docs.aws.amazon.com/STS/latest/UsingSTS/Welcome.html
http://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Introduction.html
http://docs.aws.amazon.com/amazondynamodb/latest/APIReference/Welcome.html
https://github.com/
http://www.slideshare.net/clifflu666/case-study-on-aws-javascript-sdk-and-dynamodb

Recommended

DynamoDB コトハジメ (2012-03-03 JAWS SUMMIT2012 )
Technology
AWS Black Belt Techシリーズ Amazon DynamoDB
Technology
with Corona SDK and Lua Game Development Corona SDK and Lua @SergeyLerg Spiral Code Studio spiralcodestudio.com Who is that guy on the stage? PHP/JavaScript/HTML C Python PHP Lua Go
Documents
DAVI FRANCELINO AUXÍLIO AO APRENDIZADO VIRTUAL … · JSON JavaScript Object Notation LMS Learning Management System PDA Personal Digital Assistant SDK Software ... apresentar os
Documents
SDK PALESTRA
Technology
Dynamodb Presentation
Technology
AWS Black Belt Tech シリーズ 2015 - Amazon DynamoDB
Technology
【JavaScript SDK ver.2】 MonacaとmBaaSでwebコンテンツのスマホアプリ化を体験しよう 20160129ver
Engineering
BIXOLON I EUROPE Deutsch · • Xamarin SDK (nur Windows, Android) • Android/iOS/Windows SDK • Linux SDK • Chrome SDK • Xamarin SDK (nur Windows, Android) • Android/iOS/Windows
Documents
DynamoDB - Uma Introdução
Technology
HyWAI for Web of Things - World Wide Web Consortium · Javascript HyWAIAPIs Native Application Created using the Android SDK orXcode. Automated using HyWAI Web Browser Webkiton iPhone
Documents
RealSense SDK
Technology
코로나 SDK 소개 :: Corona SDK
Education
DynamoDB Media Server (2012-06-12 DynamoDB+EMR ハンズオン)
Technology
【JavaScript SDK ver.2】MonacaとmBaaSでプッシュ通知を体験しよう(for Android & iOS)
Engineering
BrainWarsのアーキテクチャ（OpsWorks & DynamoDB編）
Technology
SDK Instruction
Documents
PRD-026 - Amazon DynamoDB
Technology
[AWSマイスターシリーズ] Amazon DynamoDB
Technology
QlikView SDK
Documents