Copyright©2014 NTT corp. All Rights Reserved.

loggregator update

尾尻 健NTT Software Innovation Center

2Copyright©2014 NTT corp. All Rights Reserved.

@kenojiri

• CloudFoundry をお仕事にして１年３ヶ月目

• 拠点は武蔵野、現場は田町

• 最近の読み物『チーム開発実践入門』技術評論社

• 広島カープ、今年こそ V

3Copyright©2014 NTT corp. All Rights Reserved.

【復習】 loggregator ってなに？

• CloudFoundry のコンポーネントの１つ• log aggregator →→ loggregator• CloudFoundry にアプリを登録・起動したり

稼働させたログを収集したり、 HTTP(WebSocket) 経由で取り出しできたり とかしてくれるコンポーネント• アプリ登録・起動時： CF 内部処理のログ• アプリ稼働中：アプリからの標準出力・標準エ

ラー出力と、 HTTP アクセスログ

以前の戦いhttp://www.slideshare.net/morika_t/16-cloud-foundry-loggregator

4Copyright©2014 NTT corp. All Rights Reserved.

loggregator の活躍っぷり (1)$ cf push hello -n helloCreating app hello in org test / space test as admin...OK

Using route hello.10.244.0.34.xip.ioOK

Uploading hello...Uploading app files from: /home/ntt/GO/src/github.com/tamac-io/cf-acceptance-tests/assets/hello-worldUploading 894, 5 filesOK

Starting app hello in org test / space test as admin...OK-----> Downloaded app package (212K)-----> Compiling Ruby/Rack-----> Using Ruby version: ruby-2.0.0-----> Installing dependencies using 1.5.2 Running: bundle install --without development:test --path vendor/bundle --binstubs vendor/bundle/bin -j4 --deployment Using bundler (1.5.2) Installing rack (1.5.2) Updating files in vendor/cache Your bundle is complete! Gems in the groups development and test were not installed. It was installed into ./vendor/bundle Bundle completed (0.40s) Cleaning up the bundler cache.-----> WARNINGS: You have not declared a Ruby version in your Gemfile. To set your Ruby version add this line to your Gemfile: ruby '2.0.0' # See https://devcenter.heroku.com/articles/ruby-versions for more information. No Procfile detected, using the default web server (webrick) https://devcenter.heroku.com/articles/ruby-default-web-server

-----> Uploading droplet (13M)

1 of 1 instances running

App started

Showing health and status for app hello in org test / space test as admin...OK

requested state: startedinstances: 1/1usage: 256M x 1 instancesurls: hello.10.244.0.34.xip.io

state since cpu memory disk#0 running 2014-05-17 11:57:30 PM 0.0% 20.1M of 256M 0 of 1G

このへんとか

5Copyright©2014 NTT corp. All Rights Reserved.

loggregator の活躍っぷり (2)$ cf logs helloConnected, tailing logs for app hello in org test / space test as admin...

2014-05-18T00:08:04.95+0900 [RTR] OUT hello.10.244.0.34.xip.io - [17/05/2014:15:08:04 +0000] "GET / HTTP/1.1" 200 13 "-" "curl/7.22.0 (x86_64-pc-linux-gnu) libcurl/7.22.0 OpenSSL/1.0.1 zlib/1.2.3.4 libidn/1.23 librtmp/2.3" 10.244.0.34:41875 vcap_request_id:e6e7eb70-50ed-4a59-6c72-91122d3cf1fc response_time:0.141390983 app_id:0e974c86-346e-4e4a-8977-105211a9586c2014-05-18T00:08:10.07+0900 [RTR] OUT hello.10.244.0.34.xip.io - [17/05/2014:15:08:09 +0000] "GET /hoge HTTP/1.1" 200 13 "-" "curl/7.22.0 (x86_64-pc-linux-gnu) libcurl/7.22.0 OpenSSL/1.0.1 zlib/1.2.3.4 libidn/1.23 librtmp/2.3" 10.244.0.34:41601 vcap_request_id:3c1095aa-b0e6-4a3d-44f0-4575f246dfba response_time:0.118677522 app_id:0e974c86-346e-4e4a-8977-105211a9586c^c

$ cf logs hello --recentConnected, dumping recent logs for app hello in org test / space test as admin...

2014-05-17T23:57:28.40+0900 [DEA] OUT Starting app instance (index 0) with guid 0e974c86-346e-4e4a-8977-105211a9586c2014-05-17T23:57:30.13+0900 [App/0] ERR [2014-05-17 14:57:30] INFO WEBrick 1.3.12014-05-17T23:57:30.13+0900 [App/0] ERR [2014-05-17 14:57:30] INFO ruby 2.0.0 (2014-05-08) [x86_64-linux]2014-05-17T23:57:30.13+0900 [App/0] ERR [2014-05-17 14:57:30] INFO WEBrick::HTTPServer#start: pid=33 port=610012014-05-17T23:58:19.52+0900 [RTR] OUT hello.10.244.0.34.xip.io - [17/05/2014:14:58:19 +0000] "GET / HTTP/1.1" 200 13 "-" "curl/7.22.0 (x86_64-pc-linux-gnu) libcurl/7.22.0 OpenSSL/1.0.1 zlib/1.2.3.4 libidn/1.23 librtmp/2.3" 10.244.0.34:38142 vcap_request_id:30522106-ba66-4094-5746-d91b4f3f2146 response_time:0.170692205 app_id:0e974c86-346e-4e4a-8977-105211a9586c2014-05-18T00:08:04.95+0900 [RTR] OUT hello.10.244.0.34.xip.io - [17/05/2014:15:08:04 +0000] "GET / HTTP/1.1" 200 13 "-" "curl/7.22.0 (x86_64-pc-linux-gnu) libcurl/7.22.0 OpenSSL/1.0.1 zlib/1.2.3.4 libidn/1.23 librtmp/2.3" 10.244.0.34:41875 vcap_request_id:e6e7eb70-50ed-4a59-6c72-91122d3cf1fc response_time:0.141390983 app_id:0e974c86-346e-4e4a-8977-105211a9586c2014-05-18T00:08:10.07+0900 [RTR] OUT hello.10.244.0.34.xip.io - [17/05/2014:15:08:09 +0000] "GET /hoge HTTP/1.1" 200 13 "-" "curl/7.22.0 (x86_64-pc-linux-gnu) libcurl/7.22.0 OpenSSL/1.0.1 zlib/1.2.3.4 libidn/1.23 librtmp/2.3" 10.244.0.34:41601 vcap_request_id:3c1095aa-b0e6-4a3d-44f0-4575f246dfba response_time:0.118677522 app_id:0e974c86-346e-4e4a-8977-105211a9586c

このへん

6Copyright©2014 NTT corp. All Rights Reserved.

Router

CCDEA

しくみ

CCDEA

Router

API log

dea_logging_agent

app log(STDOUT/STDERR)

warden container

appinstance

access log

loggregatortrafficcontroller

loggregator_trafficcontroller

staging log

loggregatorloggregator

log ・・・ log

アプリユーザ CF ユーザ

UDPWebSocket

CLI

・アプリ毎にオンメモリに保持・最大保持行数は loggregator の設定値

7Copyright©2014 NTT corp. All Rights Reserved.

ログを集める UDP ルート

• Syslog プロトコルもどき• １行１メッセージ

• 各メッセージにアプリの UUID(appid) と送出元コンポーネントの種別 (API/DEA/STG/APP/RTR/…) が載っており、共有鍵 ( デプロイ時設定値 ) で署名される• github.com/cloudfoundry/

loggregatorlib/emitter を使用

• 再送はしないので、取りこぼしがあり得る

8Copyright©2014 NTT corp. All Rights Reserved.

ログを取り出す WebSocket ルート

• WebSocket のエンドポイント URL は/v2/info API で示される

$ curl -k https://api.10.244.0.34.xip.io/v2/info

{"name":"vcap", "build":"2222", "support":"http://nttlabs.info/", "version":2, "description":"Cloud Foundry on bosh-lite", "authorization_endpoint":"https://uaa.10.244.0.34.xip.io", "api_version":"2.4.0", "logging_endpoint":"wss://loggregator.10.244.0.34.xip.io:443"}

9Copyright©2014 NTT corp. All Rights Reserved.

ログを取り出す WebSocket ルート

• 実際の接続先 URL はパラメータとして appid を指定• Gorouter のアクセスログより...loggregator.10.244.0.34.xip.io:443 - [22/05/2014:15:08:04 +0000] "GET /tail/?app=031ac4c4-d2b6-4df8-a700-a2513fac8e10 HTTP/1.1" MissingResponseStatusCode 0 "-" "-" 10.244.0.34:45519 vcap_request_id:afc124a9-9891-45c7-4aba-bde23ffaed67 response_time:MissingFinishedAt app_id:...loggregator.10.244.0.34.xip.io:443 - [22/05/2014:15:24:09 +0000] "GET /recent?app=cdf44a31-f159-4458-a9ca-0929555f44dc HTTP/1.1" 400 26 "-" "Go 1.1 package http" 10.244.0.34:51756 vcap_request_id:5f55da2a-e2ea-44b3-5c38-faf763ab027a response_time:0.512795756 app_id:...loggregator.10.244.0.34.xip.io:443 - [22/05/2014:15:24:10 +0000] "GET /dump/?app=cdf44a31-f159-4458-a9ca-0929555f44dc HTTP/1.1" MissingResponseStatusCode 0 "-" "-" 10.244.0.34:51765 vcap_request_id:5e2d2fde-ac58-4636-5c66-16e239189671 response_time:MissingFinishedAt app_id:...

10Copyright©2014 NTT corp. All Rights Reserved.

log の貯めっぷりを例えるなら

• sink• アプリ毎に sink ができる

• 新しい appid のログが来たら新たな sink をつくる• ある appid のログが最近来なくなったら sink を消す

• ログは水。 sink の容量には限りがある• 古いログからあふれる

• たまっているログは dump できる• 全 dump も最近の？？行だけ dump もできる

・・・

app 1 app 2 app N

11Copyright©2014 NTT corp. All Rights Reserved.

もう一つのログ取り出しルート

syslog drain

app 1

WebSocket クライアント

WebSocket ルート・・・ sink につなぎこむ

syslog 受信サーバ(rsyslogd, Splunk, fluentd, …)

syslog drain ルート・・・排出先を指定する

どうやって？

12Copyright©2014 NTT corp. All Rights Reserved.

syslog drain 指定の方法

• CLI から指定可能

　　　※ URL の scheme は、 syslog syslog-tls https を使用可

• 例えば自前の fluentd に突っ込む場合$ cf create-user-provided-service my-fluentd -l syslog://fluentd.hoge.me:5140$ cf push my-app$ cf bind-service my-app my-fluentd

$ cf create-user-provided-service <drain 名 > –l <syslog drain URL>

13Copyright©2014 NTT corp. All Rights Reserved.

これからの loggregator

• Diego との連携• etcd に保持されているアプリ情報の活用

→ アプリのライフサイクルをリアルタイムで把握し、　 log sink の生成・消去タイミングを早める

DEADiego

warden container

appinstance

loggregatortrafficcontroller

loggregator_trafficcontrollerstaging log

app log

loggregatorloggregator

log sink ・・・ log sink

etcd クラスタ

14Copyright©2014 NTT corp. All Rights Reserved.