Upload
cliff-chao-kuan-lu
View
609
Download
3
Embed Size (px)
DESCRIPTION
Case study on how to manipulate AWS DynamoDB as well as IAM / STS with their JavaScript SDK in the Browser. I keep notes in memo and comments a lot, so please download and read it if you're really interested. It's a powerpoint, and if that's a problem, please let me know. I'll try convert it to PDF or some other open / free formats. Licensed under CC-BY / MIT (demo project).
Citation preview
AWS JavaScript SDK
and DynamoDBCliff Chao-kuan Lu
A Case Study:
May 21st ‘14, AWS User Group Taiwan
04/07/2023CC-BY 3.0, Cliff Chao-kuan Lu <[email protected]>
2
授權與格式說明 以超連結 (hyperlink) 代替引用 (attribution)
引用外部內容均◦已取得授權,或◦包含原始連結,並在合理範圍內引用
本文件原創內容以 CC-BY 3.0 釋出
04/07/2023CC-BY 3.0, Cliff Chao-kuan Lu <[email protected]>
3
About cliffluFull-stack Web DeveloperAWS Solutions ArchitectNerd
about.me/clifflu
04/07/2023CC-BY 3.0, Cliff Chao-kuan Lu <[email protected]>
4
About cliffluFull-stack Web DeveloperAWS Solutions ArchitectNerd
about.me/clifflu
Level Up !! Professional
04/07/2023CC-BY 3.0, Cliff Chao-kuan Lu <[email protected]>
5
Intro◦AWS JS SDK◦IAM◦DynamoDB
Case Study◦Headless Poller
大綱
04/07/2023CC-BY 3.0, Cliff Chao-kuan Lu <[email protected]>
7
AWS SDK for JavaScript in the Browser
這名字好長
Part 1.1
04/07/2023CC-BY 3.0, Cliff Chao-kuan Lu <[email protected]>
9
官網 起自 AWS SDK for Node.js2.0 更名並支援 Browser
AWS JS SDK / Browser
04/07/2023CC-BY 3.0, Cliff Chao-kuan Lu <[email protected]>
11
AutoScaling, CloudFormation, CloudFront, Cloudsearch, CloudTrail, CloudWatch, DataPipeline, DirectConnect, DynamoDB, EC2, ElastiCache, ElasticBeanstalk, ElasticTranscoder, ELB, EMR, Glacier, IAM, ImportExport, Kinesis, OpsWorks, RDS, Redshift, Route53, S3, SES, SimpleDB, SNS, SQS, StorageGateway, STS, Support, SWF
服務
04/07/2023CC-BY 3.0, Cliff Chao-kuan Lu <[email protected]>
15
SDK for◦PHP, Python, Node.js◦Java, .NET, ◦Ruby
AWS SDK for Android & iOSDec. 8th, 2010
親友
04/07/2023CC-BY 3.0, Cliff Chao-kuan Lu <[email protected]>
18
機房、機器、網路、電力很難搞交給 Amazon Web Services 正好 那 EC2 Instance 呢?
反思
04/07/2023CC-BY 3.0, Cliff Chao-kuan Lu <[email protected]>
19
EC2 各種麻煩◦Load Balancing◦Types◦Contracts◦AutoScaling Parameters Pattern
淵藪
04/07/2023CC-BY 3.0, Cliff Chao-kuan Lu <[email protected]>
20
太多 Gotcha 每層服務、各層之間都要考慮 複雜度可能變 M x N
Scaling & HA
04/07/2023CC-BY 3.0, Cliff Chao-kuan Lu <[email protected]>
21
範例example.com
ELBRoute53
CF
S3StaticConte
nt
SharedEnv
Auto Scaling group
AMI
AZ 2Web
ServersS1
Secondary
S2 Secondary
Config
AZ 1Web Servers
S1 Primary
S2 Primary
Config
AZ 3Web Servers
Config + Arbitor
mongod
04/07/2023CC-BY 3.0, Cliff Chao-kuan Lu <[email protected]>
23
促成符合 SOA Pattern 之架構優點Web Page
HTML
CSSJS
Authenticate &
Authorize
Services
1. Auth Request
3. Authorized Identity
2. Access Token
04/07/2023CC-BY 3.0, Cliff Chao-kuan Lu <[email protected]>
24
容易整合其他服務 IAM
STS: Security Token Service WIF: Web Identity Federation
DynamoDB, S3, … 自有服務, SOA 嘛 O.o/
優點
04/07/2023CC-BY 3.0, Cliff Chao-kuan Lu <[email protected]>
27
RESTful, SOAP (deprecated)Dev Tools 是好朋友API
04/07/2023CC-BY 3.0, Cliff Chao-kuan Lu <[email protected]>
29
Identity and Access Management
AWS 權限樞紐
Part 1.2
04/07/2023CC-BY 3.0, Cliff Chao-kuan Lu <[email protected]>
30
Root UserGroup / IAM UserRoles
◦AWS SVC◦X-Account◦IdP Web SAML
IAM: Identity
04/07/2023CC-BY 3.0, Cliff Chao-kuan Lu <[email protected]>
31
Password MFA (Multi-factor authentication)
Access / Secret Key Pair X.509 certificate
3rd Party ◦SAML◦Web Identity Federation
IAM: Authentication
04/07/2023CC-BY 3.0, Cliff Chao-kuan Lu <[email protected]>
33
IAM: Policy Effect: Deny | Allow Action:
◦允許呼叫的 API Resource:
◦arn Principal
◦授權端限制 Condition
◦其他限制
04/07/2023CC-BY 3.0, Cliff Chao-kuan Lu <[email protected]>
34
for rule in rules:◦Explicit Deny -> Deny◦Explicit Allow -> Allow◦Default Deny
稽核
04/07/2023CC-BY 3.0, Cliff Chao-kuan Lu <[email protected]>
35
IAM: arn
格式◦冒號分隔◦首二節固定為 arn:aws◦Service◦Region◦Account◦Resource Identifier
04/07/2023CC-BY 3.0, Cliff Chao-kuan Lu <[email protected]>
36
給路人甲的◦Access / Secret Key Pair◦Management Console 登入權限
IAM : STS
04/07/2023CC-BY 3.0, Cliff Chao-kuan Lu <[email protected]>
38
Web Identity Federation
Facebook, Google 可也
04/07/2023CC-BY 3.0, Cliff Chao-kuan Lu <[email protected]>
39
只接受下列驗證機制◦表三家: Amazon, Facebook, Google◦SAML 說明列表 IAM Partners
限制
04/07/2023CC-BY 3.0, Cliff Chao-kuan Lu <[email protected]>
40
Trust Relationships◦Identity Provider◦Client ID
Permissions
IAM Role for WIF
04/07/2023CC-BY 3.0, Cliff Chao-kuan Lu <[email protected]>
41
用戶可透過第三方驗證與 IAM:STS, WIF 授予調用 AWS API 之權限
至此
04/07/2023CC-BY 3.0, Cliff Chao-kuan Lu <[email protected]>
42
WIF PlaygroundLogin with amazonAWS Documentation
◦Using IAM◦Using STS◦SDK for JavaScript
參照
04/07/2023CC-BY 3.0, Cliff Chao-kuan Lu <[email protected]>
43
DynamoDBManaged NoSQL Service
Part 1.3
04/07/2023CC-BY 3.0, Cliff Chao-kuan Lu <[email protected]>
44
Managed NoSQL Service 取代 SimpleDB 三本柱
◦Scalable◦Available◦Fast
DynamoDB
04/07/2023CC-BY 3.0, Cliff Chao-kuan Lu <[email protected]>
45
Data ModelScalar
◦Number{“N”: “300”}
◦String{“S”: “300”}
◦Binary{“B”: “BASE64”}
Multi-valued◦Number Set
{“NS”: [“1”,”2”,”3.14”]}◦String Set
{“SS”: [“A”,”b”]}◦Binary Set
{“BS”: [“BASE64”]}
04/07/2023CC-BY 3.0, Cliff Chao-kuan Lu <[email protected]>
46
Document size: 64 KB (UTF-8) 5 LSIs / 5 GSIs per Table Min throughput: 1
Hash Key: 2 KB Range Key: 1KB
BatchGetItem: 1MB or 100 items BatchWriteItem: 1MB or 25 items
Limits
04/07/2023CC-BY 3.0, Cliff Chao-kuan Lu <[email protected]>
47
Data Storage:◦約是 S3 10x
Provisioned Throughputs◦Reads : 4kb 循序可合併 Eventually Consistent 消耗減半
◦Writes : 1kb
Pricing
04/07/2023CC-BY 3.0, Cliff Chao-kuan Lu <[email protected]>
48
Hash Key [opt] Range Key
Primary Key
hash: “a”
hash: “b”
{“hash”: “a”, “range”: “123”, …}{“hash”: “a”, “range”: “223”, …}{“hash”: “a”, “range”: “321”, …}
{“hash”: “b”, “range”: “3”, …}{“hash”: “b”, “range”: “22”, …}{“hash”: “b”, “range”: “321”, …}
04/07/2023CC-BY 3.0, Cliff Chao-kuan Lu <[email protected]>
49
Primary Key
hash: “a”
hash: “b”
{“hash”: “a”, “range”: “123”, …}{“hash”: “a”, “range”: “223”, …}{“hash”: “a”, “range”: “321”, …}
{“hash”: “b”, “range”: “3”, …}{“hash”: “b”, “range”: “22”, …}{“hash”: “b”, “range”: “321”, …}
SortedSharded
04/07/2023CC-BY 3.0, Cliff Chao-kuan Lu <[email protected]>
50
Index Name
Hash KeyRange KeyProjection
Shared Throughputs
Local Secondary Index
04/07/2023CC-BY 3.0, Cliff Chao-kuan Lu <[email protected]>
51
Index Name / Hash / Range Key ◦比照 LSI◦不要求 uniqueness
有自己的 Throughputs
Eventual ConsistencyProjected Attributes
Global Secondary Index
04/07/2023CC-BY 3.0, Cliff Chao-kuan Lu <[email protected]>
52
針對特定 Index (PK, LSI or GSI) 查詢 支援 Condition 支援 Filter 高效
Query
04/07/2023CC-BY 3.0, Cliff Chao-kuan Lu <[email protected]>
53
現適用於 scan 及 query 消耗 throughput 不變 減低 DynamoDB <-> Caller 傳輸boto 僅 boto.dynamodb2 支援
Filter
04/07/2023CC-BY 3.0, Cliff Chao-kuan Lu <[email protected]>
55
Upsert 需包含 Primary Key
Update: PATCHPut: POST (PUT)
支援 Conditional Operation
Put, Update
04/07/2023CC-BY 3.0, Cliff Chao-kuan Lu <[email protected]>
56
支援 Conditional Operation 用於刪除 Item
若要刪除 Attribute ,需使用 updateItem
Delete
04/07/2023CC-BY 3.0, Cliff Chao-kuan Lu <[email protected]>
57
“Expected” 可實現 MVCC pattern
配合 Document Atomicity 可模擬transactional behavior (2-Phase Commit)
Conditional Operation
04/07/2023CC-BY 3.0, Cliff Chao-kuan Lu <[email protected]>
58
updateItem◦AttributeUpdates Value Action:
PUT DELETE ADD
Atomic Increment
04/07/2023CC-BY 3.0, Cliff Chao-kuan Lu <[email protected]>
60
Fine-Grained Policies
Item◦“dynamodb:LeadingKeys”: [“xxx”]
Attribute◦“dynamodb:Attributes”: [“xxx”,”yyy”]
04/07/2023CC-BY 3.0, Cliff Chao-kuan Lu <[email protected]>
61
While True: Item.save(expect=…)
1unit for Read / Write
More on Throughput
04/07/2023CC-BY 3.0, Cliff Chao-kuan Lu <[email protected]>
62
Stats from DynamoDB
More on Throughput
04/07/2023CC-BY 3.0, Cliff Chao-kuan Lu <[email protected]>
63
CloudWatch 顯示低階數據`Expect` 不消耗 Read Unit 允許 short burst
猜想
04/07/2023CC-BY 3.0, Cliff Chao-kuan Lu <[email protected]>
64
了解 DynamoDB 的◦設計目標◦調校 Index Throughput
◦操作◦計費
小結
04/07/2023CC-BY 3.0, Cliff Chao-kuan Lu <[email protected]>
66
Phil: clifflu 你要不要講五月小聚
Henry: 聽說 Cloudflare + wordpress 會爆炸clifflu: 好啊,就講這個吧
緣起
04/07/2023CC-BY 3.0, Cliff Chao-kuan Lu <[email protected]>
67
秒殺WP 看 location.href 重導頁面 講不滿三十分鐘 ˇˇ 只好調出備用題目
然後
04/07/2023CC-BY 3.0, Cliff Chao-kuan Lu <[email protected]>
68
EC2 各種麻煩◦Load Balancing◦Types◦Contracts◦AutoScaling Parameters Pattern
回想
04/07/2023CC-BY 3.0, Cliff Chao-kuan Lu <[email protected]>
70
使用服務◦IAM, S3 (DynamoDB), CF
Octopress / Jekyll !?Clone Ruby 好像很遜 >///<
寫個 Blog system ?
04/07/2023CC-BY 3.0, Cliff Chao-kuan Lu <[email protected]>
72
前端◦Angular.js 潮◦OAuth 勁
後端◦從缺,帥
資料◦DynamoDB ,猛
線上投票
04/07/2023CC-BY 3.0, Cliff Chao-kuan Lu <[email protected]>
73
前端◦Angular.js 潮◦OAuth 勁
後端◦從缺,帥
資料◦DynamoDB ,猛
線上投票需要 https
CloudFront
04/07/2023CC-BY 3.0, Cliff Chao-kuan Lu <[email protected]>
74
申請 APP◦Amazon◦Facebook◦Google
將 access_token 透過 STS 轉換為 Access / Secret Key Pair
第三方授權
04/07/2023CC-BY 3.0, Cliff Chao-kuan Lu <[email protected]>
75
欄位: uid, q_id, o_id
Primary Key◦(uid, q_id) : unique 確保每人每題一票◦選 uid 為 hash key Authentication Cardinality
DynamoDB
04/07/2023CC-BY 3.0, Cliff Chao-kuan Lu <[email protected]>
76
updateItem (省略 callback)投票
Primary Key
Upsert
Return on Update
04/07/2023CC-BY 3.0, Cliff Chao-kuan Lu <[email protected]>
78
正解: Worker ,但霸氣不足 需求:
◦uid 不外漏◦affordable◦快
計票
04/07/2023CC-BY 3.0, Cliff Chao-kuan Lu <[email protected]>
79
建立 (q_id, o_id) 之 GSI禁止讀取 uid 欄位 取出列數即為總票數 循序讀取,節省 read capacity
計票: GSI
04/07/2023CC-BY 3.0, Cliff Chao-kuan Lu <[email protected]>
80
LSI / GSI 必定包含 Primary Key 透過 Query / Scan 取得 Item 時,必須允許讀取 primary key
麻煩,天大的麻煩
04/07/2023CC-BY 3.0, Cliff Chao-kuan Lu <[email protected]>
82
運用 “ select”: “COUNT” 只計算票數,不取 Item body
循序讀取
計票: GSI
04/07/2023CC-BY 3.0, Cliff Chao-kuan Lu <[email protected]>
83
Document 沒寫Boto 沒使用
◦dynamodb 實做了 query.count() ,透過取回 item 記數◦dynamodb2 未有類似功能
霸氣
04/07/2023CC-BY 3.0, Cliff Chao-kuan Lu <[email protected]>
88
參數Throughpu
tsRead Write
Table 1 5GSI 19 5
04/07/2023CC-BY 3.0, Cliff Chao-kuan Lu <[email protected]>
89
參數Throughpu
tsRead Write
Table 1 5GSI 19 5
Total #Reads8kb data 消耗 1
投票消耗GSI ~ Table
~6.6 USD/mo
04/07/2023CC-BY 3.0, Cliff Chao-kuan Lu <[email protected]>
90
改用 Batched Query ?◦不支援 select COUNT
優化流程◦資料更新◦後台計票◦操作介面◦PR please
改進?
04/07/2023CC-BY 3.0, Cliff Chao-kuan Lu <[email protected]>
93
/clifflu/headless-poller
簡報下載: AWS Doc
◦IAM◦STS◦DynamoDB
API
Links