Node.js 與 Google Cloud Storage多麼痛的領悟

關於我

• Ian Wu

• 瘋⼈人院院⻑⾧長

• 頑⽪皮⼯工坊 Backend Engineer

• http://blog.ianwu.tw/about-me/

Why google cloud storage• 內建 CDN

• Google Cloud Storage behaves essentially like a Content Delivery Network (CDN) with no work on your part because publicly readable objects are, by default, cached in the Google Cloud Storage network.

• try try see

• try 到死

• 有 USD 500 的 credit 

OAuth2• JWT (JSON Web Token)

• Google Cloud console

• credential: service account

• covert p12 > pem

Authentication - Google Cloud Storage — Google Cloud Platform https://cloud.google.com/storage/docs/authentication#service_accounts

OAuth2• Get token

• payload

• jwt sign

{ iss: '460520686343-k6tfn73sentmh0ss5nu67kniorbcta8n@developer.gserviceaccount.com', scope: 'https://www.googleapis.com/auth/devstorage.full_control', aud: 'https://accounts.google.com/o/oauth2/token', exp: 1418280623, iat: 1418280563 }

Get Google JWT token. https://gist.github.com/onlinemad/28341a343ecde186a410

// sign with RSA SHA256 var cert = fs.readFileSync('google_cloud_key.pem'); // get private key var claim = jwt.sign(payload, cert, { algorithm: 'RS256' });

OAuth2• token

• 使⽤用 token

{ access_token: 'ya29.2QA9sZg_YtCTGJf1d6Vzxr_4ypioiaIdHJBmgxq6b1HsJuAPODCHnCvt', token_type: 'Bearer', expires_in: 3600 }

headers: { Authorization: 'Bearer ' + token.access_token }

Upload URI• Upload URI, for media upload requests

• upload/storage/v1/b/bucket/o

• Metadata URI, for metadata-only requests:

• storage/v1/b/bucket/o

• APIs Explorer currently supports metadata requests only.

Upload method• simple

• 就 post 上傳檔案

• multipart（推薦使⽤用）

• 可以連 metadata ⼀一起上傳

• request 某⼀一個版本以上才有⽀支援

• resumable

• 沒⽤用過

• node-youtube-resumable-uploadhttps://github.com/grayleonard/node-youtube-resumable-upload

multipart• request

var url = 'https://www.googleapis.com/upload/storage/v1/b/yourbucket/o?' + qs.stringify(querystring); request.post({ preambleCRLF: true, postambleCRLF: true, url: url, multipart: [ { 'Content-Type': 'application/json', body: JSON.stringify(metadata) }, { body: __newFile } ], headers: { Authorization: 'Bearer ' + token.access_token } });

multipart• body

• query string

• 不能跟 Request body ⼀一起⽤用

{ cacheControl: 'public, max-age=604800', acl: [{ entity: 'allUsers', role: 'READER' }, { entity: 'project-owners-692227494718', role: 'OWNER' }] }

Directory structure• ⼀一切都是平的

• 跟 s3 ⼀一樣

• 所以沒有建⽴立 folder 這件事情

• name = foo/bar.jpg;

Directory structure• simple

• /o?name=foo%2Fbar.jpg

• multipart

• body.name = foo/bar.jpg

Access URL• Standard（推薦）

• storage.googleapis.com/<bucket>/<object>

• <bucket>.storage.googleapis.com/<object>

• CNAME

• travel-maps.example.com CNAME c.storage.googleapis.com

• no ssl

• Cookie-based Authentication

• 沒⽤用過

Versioning• 預設是關掉的

• qs + generation

➜ ~ gsutil versioning get gs://onlinemad-versioning gs://onlinemad-versioning: Suspended ➜ ~ gsutil versioning set on gs://onlinemad-versioning Enabling versioning for gs://onlinemad-versioning/... ➜ ~

{ "kind": "storage#object", "id": "onlinemad-dev/uploaded.jpg/1418291876469000", "selfLink": "https://www.googleapis.com/storage/v1/b/onlinemad-dev/o/uploaded.jpg", "name": "uploaded.jpg", "bucket": "onlinemad-dev", "generation": "1418291876469000", "metageneration": "1", "contentType": "image/jpeg", "updated": “2014-12-11T09:57:56.468Z”, }

ACL[ { "entity": "project-owners-460520686343", "projectTeam": { "projectNumber": "460520686343", "team": "owners" }, "role": "OWNER" }, { "entity": "project-editors-460520686343", "projectTeam": { "projectNumber": "460520686343", "team": "editors" }, "role": "OWNER" }, { "entity": "project-viewers-460520686343", "projectTeam": { "projectNumber": "460520686343", "team": "viewers" }, "role": "READER" }, { "entity": "user-00b4903a9745459d3abf193213c0f30d5dea50ee7e3e318007a7edfaecb646e5", "entityId": "00b4903a9745459d3abf193213c0f30d5dea50ee7e3e318007a7edfaecb646e5", "role": "OWNER" } ]

ACL• 我需要 public read

• 所以 request.post({ preambleCRLF: true, postambleCRLF: true, url: url, multipart: [{ 'Content-Type': 'application/json', body: JSON.stringify({ name: 'acl_multipart_upload_public_read.jpg', acl: [{ entity: 'allUsers', role: 'READER' }] }) }, { body: data }], headers: { Authorization: 'Bearer ' + token.access_token } })

ACL

➜ ~ gsutil acl get gs://onlinemad-dev/acl_simple_upload_public_read.jpg AccessDeniedException: Access denied. Please ensure you have OWNER permission on gs://onlinemad-dev/acl_simple_upload_public_read.jpg.

這是 feature 不是 bug 這是 feature 不是 bug 這是 feature 不是 bug

ACL

ACLrequest.post({ preambleCRLF: true, postambleCRLF: true, url: url, multipart: [{ 'Content-Type': 'application/json', body: JSON.stringify({ name: 'acl_multipart_upload_public_read_add_owner.jpg', acl: [{ entity: 'allUsers', role: 'READER' }, { entity: 'project-owners-460520686343', role: 'OWNER' }] }) }, { body: data }], headers: { Authorization: 'Bearer ' + token.access_token } })

我的領悟

– Ian Wu

「還沒有⼈人分享 Google Service 時， 請勿輕易嘗試」

– Ian Wu

「當你試了 Google Service 時， 請來分享」

謝謝⼤大家