1

Logging infrastructure in PaaS with Fluentd

Rakuten. Inc,　22/08/2012

Yohei Sasaki / Waldemar Quevedo Paas Dev&Ops team

2

About us

§  ワリと申します。メキシコ人です。 (Wally, from Mexico)

§  Living in Tokyo since 2010.

§  Member of the PaaS Dev&Ops team at Rakuten, programming in Ruby.

§  Emacs user (org-mode rocks, try .org markup in Github)

§  OpenStreetMap (Mapnik rocks)

§  ささきと申します。日本人です。 (Sasaki, from Japan)

§  Living in Japan since 1982.

§  Tech Lead of the PaaS Dev&Ops team at Rakuten, Architect of Rakuten PaaS.

§  Node.js user / node-fluentd-logger

§  CouchDB, …etc

3

Agenda

§  Cloud Foundry

§  Cloud Foundry + Fluentd §  Fluentd Pros & Cons §  Misc

4

Cloud Foundry

•  Open Application Platform as a Service project

•  By VMware

•  Written in Ruby

•  OpenSource http://github.com/cloudfoundry/vcap •  Hosted version http://www.cloudfoundry.com/

5

Why Cloud Foundry?

•  Last year it was decided to introduce Cloud Foundry to build a PaaS in Rakuten

http://techtarget.itmedia.co.jp/tt/news/1111/28/news02.html http://www.slideshare.net/rakutentech/cloud-foundry-10423477 •  Main points:

–  Open Source –  Separation of concerns from IaaS –  Scalable architecture –  Multi language

6

What Cloud Foundry does?

•  Easy to deploy applications!!

$ mkdir app.js $ vi app.js $ vmc push yssk22-myapp $ vmc instances yssk22-myapp 3

•  Not easy to deploy business ready applications yet. –  We are now solving our own issues by modifying Cloud Foundry source

–  One critical thing is Logging!

–  Now we’ve solved this matter basically, let’s share our case study.

7

Current logging issue in Cloud Foundry

$ vmc push yssk22-myapp!!$ curl http://yssk22-myapp.cloudfoundry.com/!!$ vmc logs yssk22-myapp!![Wed, 22 Aug 2012 08:03:48 GMT] Hello World! !!$ vmc update yssk22-myapp!!$ vmc logs yssk22-myapp!

8

Current logging issue in Cloud Foundry

$ vmc push yssk22-myapp!!$ curl http://yssk22-myapp.cloudfoundry.com/!!$ vmc logs yssk22-myapp!![Wed, 22 Aug 2012 08:03:48 GMT] Hello World! !!$ vmc update yssk22-myapp!!$ vmc logs yssk22-myapp!

Oops, logs are gone.

9

Persisting logs is a must for business applications!

•  Monitoring •  Analysis •  Visualization

Ask a data scientist =>

http://www.slideshare.net/doryokujin/hadoop-and-the-data-scientist

Sorry we lost everything :P

10

Cloud Foundry does not persists the logs from apps…

•  Enter Fluentd!

11

Brief overview of Cloud Foundry architecture

Source: http://www.slideshare.net/derekcollison/design-of-cloud-foundry

Droplet

Execution

Agent

12

Approach

DEA

app

app

fluentd

fluentd

DEA

app

app

fluentd

fluentd

Attach a Fluentd collector to one application instance upon dispatch.

Log Storage

13

Collecting the logs from the apps

•  Log collector monitors changes in the files –  Stdout, stderr logs in general –  Catalina, manager, host-manager logs for Java –  error.log for PHP apps

•  Also makes available:

–  VCAP_LOG_PORT env variable to flush logs directly

–  VCAP_LOG_STREAM_PORT to inspect the logs in realtime

DEA

app

app

fluentd

fluentd

app fluentd

app fluentd

app fluentd

14

Collecting the logs from the apps

DEA

app fluentd

Fluent::Logger => VCAP_LOG_PORT

Sends logs directly to Fluentd making unnecessary having to write to file. VCAP_LOG_STREAM_PORT

Streams the logs to a client {‘type’: ‘serverLog’, ‘text’: ‘Hello World’, ‘level’: ‘INFO’}

Log Storage

15

Small setup for collecting the logs

Log Storage Server

fluentd Rack app

CloudController

Collected log files

DEA

app

app

fluentd

fluentd

DEA

app

app

fluentd

fluentd

Log Storage Server

Collected log files

/logs/:path

CloudController

/logs/:path

Rack app

16

How is it setup?

Attached to an application instance on startup by DEA

•  (1) and (2) is a “stat file” used by the input plugin, which allows it to pick up from the last position it had monitored.

•  (3) is the configuration of the plugins dynamically

generated by the DEA •  (4) and (5) are the scripts executed to manage the

fluentd process •  (6) and (7) are the supervisor and main process

ids from Fluentd •  (8) is where any logs produced by the logging

daemon are stored

app/ !

fluentd/!

|-- stderr.stat (1)!

|-- stdout.stat (2)!

|-- fluentd.conf (3)!

|-- startup (4)!

|-- stop (5)!

fluentd_child.pid (6)!

fluentd.pid (7)!

logs/!

|-- fluentd.log (8)!

|-- stdout.log!

|-- stderr.log!

ruby!

run.pid!

startup!

stop!

17

fluentd.conf :: Input configuration

!

<source>!

type forward!

port 12345!

</source>!

!

<source>!

type cf_app_logs!

instance_dir /var/vcap.local/dea/apps/appname-0-3e9676485562c64d81a61de2df05903a!

runtime ruby19!

framework sinatra!

tag logs.storage!

</source>!

!

!

Set up VCAP_LOG_PORT to flush the logs directly

Specify runtime and dir to monitor files

18

fluentd.conf :: Output configuration

<match logs.**>!

type copy!

<store>!

type cf_app_forward!

flush_interval 5s!

buffer_type file!

buffer_path fluentd/fluentd_buffer!

!

cf_app_user foo@example.com!

cf_app_name appname!

cf_app_instance_id 3e9676485562c64d81a61de2df05903a!

cf_app_instance_index 0!

Settings for when forwarding to the log storage, we include user information here so that it cannot be overrided by a logging library.

19

fluentd.conf :: Output configuration

<server>!

name logserver!

host 192.168.2.1!

port 4224!

</server>!

<server>!

name backup_logserver!

host 192.168.2.2!

port 4224!

standby!

</server>!

</store>!

<store>!

type cf_app_logs_streaming!

port 43208!

</store>!

</match>!

Location of the log storage server: -  We use a active-standby setup at the

moment. -  When primary Logserver is down, logs

are forwarded to the one on standby. When both are down, fluentd buffers to files the logs locally until one of them is available.

Set up VCAP_LOG_STREAM port to enable tail –f like support.

20

…And now that we managed to collect the logs,

we also can support the following 2 commands:

21

vmc tail

22

vmc log-storage

23

What we could do

•  Persisting the logs in Cloud Foundry is a must! •  Support to persist the logs is a key issue in adoption of Cloud

Foundry for developing application services.

•  Thanks to Fluentd we could enable users to have log archives in a straightforward way.

•  Cloud Foundry and Fluentd themselves are both written in Ruby so integration was relatively easy.

24

What we could not do

•  It was difficult to let the user have access to the logs from Nginx (used as a Router in CF)

•  Then the app-v1.cf.rakuten.co.jp subdomain becomes available to other user.

•  Need to keep track that user X had Y domain at Z time for application named A in real time while storing the logs.

# Sample release process!

vmc push app-v1 --runtime ruby19 => app-v1.cf.rakuten.co.jp!

vmc map app-v1 olympics.rakuten.co.jp => Released!!

vmc push app-v2 --runtime ruby19 => app-v2.cf.rakuten.co.jp!

vmc map app-v2 olympics.rakuten.co.jp => Load balancing between apps!

vmc unmap app-v1 olympics.rakuten.co.jp => Release 2 complete.!

25

Fluentd: Pros and Cons

•  Pros –  Easy to include in Cloud Foundry compared to other solutions

•  Possible to include it like gem 'fluentd' in a Ruby component –  Easy to change storage technology depending on scale –  Easy to extend –  Growing community \o/

•  Cons –  Fluentd needs better testing tools –  Better materials about Fluentd from the community?

•  Learning curve is already low •  Just we need to share more about how we are using Fluentd •  e.g. 'The Little Fluentd Book', 'Advanced Fluentd Recipes’

–  Cool.io is not maintained

26

Misc: Playing with the message bus

•  Cloud Foundry is built with Eventmachine.

•  Fluentd is built with Cool.io.

•  Cool.io and Eventmachine do not play along very well.

•  The message bus from Cloud Foundry named NATS is also built using Eventmachine.

27

Misc: NATS input plugin

# This works, though a warning is displayed from Cool.io overriding Eventmachine!

require 'nats/client'!

!

module Fluent!

class NatsInput < Input!

Plugin.register_input('nats', self)!

def run!

NATS.start(:uri => @nats_connection_uri) do!

!

# Triggered when an url changing event occurs (remap, change)!

NATS.subscribe('dea.update') do |msg|!

Engine.emit('logs.dea_update', Time.now.to_i, msg)!

end!

!

# Triggered when an application is started or dea is started!

NATS.subscribe('dea.*.start') do |msg|!

Engine.emit('logs.dea_start', Time.now.to_i, msg)!

end!

end!

end!

28

Misc: Adding new forwarding nodes dynamically through NATS?

# DEA

@droplets.each_value do |instance|

begin

instance_collector = Fluent::Logger::FluentLogger.new('logs', instance[:logging_ip], instance[:logging_port])

instance_collector.post('storage', {'type': 'serverConf', 'host': '127.0.0.1', 'port': 23938 })

rescue => e

@logger.error "Error: #{e} -- #{e.backtrace}"

end

end

# Extended Forward plugin

module Fluent

class CloudfoundryAppForwardOutput < ForwardOutput

Plugin.register_output('cf_app_forward', self)

def emit(tag, es, chain)

es.each do |emit_time, record|

# Add a new forwarding server

if record.has_key?('type') and record['type'] == 'serverConf’

add_node(record['host'], record['port’])

next

end

29

FIN

You can enable log collection to your Cloud Foundry by merging this patch: https://github.com/rakutentech/dea/commit/64d271904de1e195cc90ca1958eadf704f530d94 Checkout our github repo for more Cloud Foundry related things: https://github.com/rakutentech/ That’s all, thanks!

ご清聴ありがとうございました！