Embed Size (px)
DESCRIPTION
Citation preview
Small Startup Fluentd
For Fluentd Beginners
@studio3104
Talk 4th Section at Fluentd Meetup #3
自己紹介
@studio3104 (Satoshi SUZUKI)
-- FluentdドリヴンでRubyはじめました
-- WEB業界2年目
-- 妻、娘、息子います
-- ソシャゲ、公式サイト、パチ(スロ)ンコ
-- こんなのやってます →→→→→→→
-- Perlと心中するらしいです
td-agent
in_forward
out_file_alternative
error log
access log
error log
access logfluent-agent-lite
access log
nginx
fluent-agent-lite
error log
out_(growthforecast|munin|zabbix)
WEB SERVERS
LOG SERVERMONITORINGSERVER
out_copy
out_parser
out_exec_filter
out_rewrite
out_forest /out_numeric_counter
out_(mail|mongo|redis|file|mysql|s3)
Mail /Store
out_copy out_exec_filter
今日お話すること
Fluentd Design Pattern的な
今日お話しないこと
具体的なConfiguration
td-agent
in_forward
out_file_alternative
error log
access log
error log
access logfluent-agent-lite
access log
nginx
fluent-agent-lite
error log
out_(growthforecast|munin|zabbix)
WEB SERVERS
LOG SERVERMONITORINGSERVER
out_copy
out_parser
out_exec_filter
out_rewrite
out_forest /out_numeric_counter
out_(mail|mongo|redis|file|mysql|s3)
Mail /Store
out_copy out_exec_filter
Recipe
fluent-agent-lite
rewrite
file-alternative
parser
numeric_counter
fluent-plugin
growthforecast
Recipe
fluent-agent-lite
rewrite
file-alternative
parser
numeric_counter
fluent-plugin
growthforecast
難易度
5もりす1あんちぽ
Recipe
fluent-agent-lite
rewrite
file-alternative
parser
forest
numeric_counter
fluent-plugin
growthforecast
Recipe
fluent-agent-lite
rewrite
file-alternative
parser
forest
numeric_counter
fluent-plugin
growthforecast
難易度
6もりす1あんちぽ
Recipe
fluent-agent-lite
rewrite
file-alternative
parser
forest
numeric_counter
fluent-plugin
growthforecast
難易度
6モリス1あんちぽ
Difficulty
is too high!!1
Recipe
fluent-agent-lite
rewrite
file-alternative
parser
forest
numeric_counter
fluent-plugin
growthforecast
難易度
6モリス1あんちぽ
Difficulty
is too high!!1でも、少しずつやれば大丈夫!!1
Recipe
fluent-plugin
プロローグ
太古
Recipe
fluent-plugin
error log
access log
error log
access log
access log
nginx
WEB SERVERS
LOG SERVER
Mail /Store
Serialize /Summarize /
Analyze
error log
cronrsync / scp
error log
access log
error log
access log
access log
nginx
WEB SERVERS
LOG SERVER
Mail /Store
Serialize /Summarize /
Analyze
error log
cronrsync / scp
Trouble!!
Need retransmission manually...
error log
access log
error log
access log
access log
nginx
WEB SERVERS
LOG SERVER
Mail /Store
Serialize /Summarize /
Analyze
error log
cronrsync / scp
Trouble!!
Need retransmission manually...
Chain!!
Recipe
fluent-agent-lite
file-alternative
fluent-plugin
第一章
生
Recipe
fluent-agent-lite
file-alternative
fluent-plugin
error log
access log
error log
access log
access log
nginx
WEB SERVERS
LOG SERVER
Mail /Store
Serialize /Summarize /
Analyze
error log
cronrsync / scp
td-agent
out_file_alternative
error log
access log
error log
access login_forwardfluent-agent-lite
access log
nginx
fluent-agent-lite
error log
WEB SERVERS
LOG SERVER
Mail /Store
Serialize /Summarize /
Analyze
scpやrsyncでまとめて転送するのと比べて何が優れているのか
・リアルタイムで転送 たとえばサーバが突然死んでも、ログサーバで直前のログを参照できる プロセス自体が死ぬと・・・
・途中でこけてもバッファリングによって再送 プロセス自体が死ぬと・・・
scpやrsyncでまとめて転送するのと比べて何が優れているのか
・リアルタイムで転送 たとえばサーバが突然死んでも、ログサーバで直前のログを参照できる プロセス自体が死ぬと・・・
・途中でこけてもバッファリングによって再送 プロセス自体が死ぬと・・・
監視は
絶対しよう
scpやrsyncでまとめて転送するのと比べて何が優れているのか
・リアルタイムで転送 たとえばサーバが突然死んでも、ログサーバで直前のログを参照できる プロセス自体が死ぬと・・・
・途中でこけてもバッファリングによって再送 プロセス自体が死ぬと・・・
監視は
絶対しよう
外道父の匠Fluentd Meetup #2 発表資料
http://blog.father.gedow.net/2012/08/23/fluentd-
meetup-vol02/
Why fluent-agent-lite?
・入れて起動したらほぼそのままでいい
・パースしない ・正規表現を通さないので効率的 ・fluent-catは入力にJSONを要求するが・・・ 「標準入力から読んでFluentdに送信するにはfluent-agent-liteが便利」 http://d.hatena.ne.jp/sfujiwara/20120611/1339373928
・対象1ファイル毎に1プロセス起動する ・これは逆に注意点でもある
Recipe
fluent-agent-lite
file-alternative
parser
forest
fluent-plugin
第二章
老害
Recipe
fluent-agent-lite
file-alternative
parser
forest
fluent-plugin
td-agent
out_file_alternative
error log
access log
error log
access login_forwardfluent-agent-lite
access log
nginx
fluent-agent-lite
error log
WEB SERVERS
LOG SERVER
Mail /Store
Serialize /Summarize /
Analyze
td-agent
out_file_alternative
error log
access log
error log
access login_forwardfluent-agent-lite
access log
nginx
fluent-agent-lite
error log
WEB SERVERS
LOG SERVER
Mail /Store
Serialize /Summarize /
Analyze
cronMidnight batch
td-agent
out_file_alternative
error log
access log
error log
access login_forwardfluent-agent-lite
access log
nginx
fluent-agent-lite
error log
WEB SERVERS
LOG SERVER
Mail /Store
Serialize /Summarize /
Analyze
cronMidnight batch Bad SQL
td-agent
out_file_alternative
error log
access log
error log
access login_forwardfluent-agent-lite
access log
nginx
fluent-agent-lite
error log
WEB SERVERS
LOG SERVER
Mail /Store
Serialize /Summarize /
Analyze
cronMidnight batch Bad SQL
Too BIG!!Loadavg
td-agent
in_forward
out_file_alternative
error log
access log
error log
access logfluent-agent-lite
access log
nginx
fluent-agent-lite
error log
WEB SERVERS
LOG SERVER
out_copy
out_parser
out_(mail|mongo|redis|file|mysql|s3)
Mail /Store
out_exec_filter
About out_exec_filter
td-agent
in_forward
out_file_alternative
error log
access log
error log
access logfluent-agent-lite
access log
nginx
fluent-agent-lite
error log
WEB SERVERS
LOG SERVER
out_copy
out_parser
out_file
out_exec_filter
access logNEO
access log
DB SERVERS
cba
ed
a
e
mysqld_multiMySQLreplicate-do-table=a,e
td-agent
in_forward
out_file_alternative
error log
access log
error log
access logfluent-agent-lite
access log
nginx
fluent-agent-lite
error log
WEB SERVERS
LOG SERVER
out_copy
out_parser
out_file
out_exec_filter
access logNEO
access log
DB SERVERS
cba
ed
a
e
mysqld_multiMySQLreplicate-do-table=a,e
#!/usr/bin/env perl
use strict;use warnings;use Data::Dumper;use DBIx::Handler;use JSON::XS;use Data::MessagePack;
$| = 1;my $mp = Data::MessagePack->new();
my $db_user = 'root';my $db_passwd = '';my $db_opt = { RaiseError => 0, PrintError => 1 };my $dsn1 = 'DBI:mysql:database=USERS1:localhost;mysql_socket=/var/lib/mysql/mysql.sock6001';my $dsn2 = 'DBI:mysql:database=USERS2:localhost;mysql_socket=/var/lib/mysql/mysql.sock6002';my $handler1 = DBIx::Handler->new( $dsn1, $db_user, $db_passwd, $db_opt );my $handler2 = DBIx::Handler->new( $dsn2, $db_user, $db_passwd, $db_opt );
while ( my $json = <STDIN> ) { my $log_column = eval { decode_json($json); }; next if ($@);
unless ( defined $log_column->{ 'x-dcmguid' || 'http_x_up_subno' || 'x-jphone-uid' } ) {
print $mp->pack($log_column);next;
}
my $sub; if ( $log_column->{'x-dcmguid'} ne '-' ) {
$sub = $log_column->{'x-dcmguid'}; } elsif ( $log_column->{'http_x_up_subno'} ne '-' ) {
$sub = $log_column->{'http_x_up_subno'}; } elsif ( $log_column->{'x-jphone-uid'} ne '-' ) {
$sub = $log_column->{'x-jphone-uid'};$sub =~ s/^.//;
} else {
print $mp->pack($log_column);next;
}
my $sql = 'SELECT uid, update_time FROM trn_user WHERE sub = ? ORDER BY update_time DESC LIMIT 1';
my $even = $handler1->dbh->selectrow_hashref( $sql, undef, $sub ); my $odd = $handler2->dbh->selectrow_hashref( $sql, undef, $sub );
my $UID = 0; if ( defined $even && defined $odd ) {
$UID = $even->{update_time} ge $odd->{update_time} ? $even->{uid} : $odd->{uid}; } elsif ( defined $even ) {
$UID = $even->{uid}; } elsif ( defined $odd ) {
$UID = $odd->{uid}; } else {
print $mp->pack($log_column);next;
}
$log_column->{message} = $log_column->{message} . ' UID' . $UID; print $mp->pack($log_column);}
About out_exec_filter
・途中の処理を得意な言語で書ける 小さいコードでいいけど、しっかり書かないとエラったり予期せず死ぬので、
運用エンジニアがコード書く練習するのに最適
・JSONとMessagePack(とTSV) ただしVer.0.10.20以降
・実行コストが高い(らしい)
・out_formatにJSONはイケてない Yajl::Parserがバッファリングするため
My template script for out_exec_filter
#!/usr/bin/env perl
use strict;use warnings;use Data::Dumper;use JSON::XS;use Data::MessagePack;
$| = 1;my $mp = Data::MessagePack->new();
while ( my $json = <STDIN> ) { my $decode = eval { decode_json($json); }; next if ($@);
unless ( defined $decode->{key} ) { print $mp->pack($decode); next; }
print $mp->pack($decode);}
Recipe
fluent-agent-lite
file-alternative
parser
forest
numeric_counter
fluent-plugin
growthforecast
第三章
革命
Recipe
fluent-agent-lite
file-alternative
parser
forest
numeric_counter
fluent-plugin
growthforecast
Recipe
fluent-agent-lite
file-alternative
parser
forest
numeric_counter
fluent-plugin
zabbix
Recipe
fluent-agent-lite
file-alternative
parser
forest
numeric_counter
fluent-plugin
munin
td-agent
in_forward
out_file_alternative
error log
access log
error log
access logfluent-agent-lite
access log
nginx
fluent-agent-lite
error log
WEB SERVERS
LOG SERVERMONITORINGSERVER
out_copy
out_parser
out_exec_filter
out_numeric_counter
out_(growthforecast|munin|zabbix)
out_(mail|mongo|redis|file|mysql|s3)
Mail /Store
out_copy out_exec_filter
td-agent
in_forward
out_file_alternative
error log
access log
error log
access logfluent-agent-lite
access log
nginx
fluent-agent-lite
error log
WEB SERVERS
LOG SERVERMONITORINGSERVER
out_copy
out_parser
out_exec_filter
out_numeric_counter
out_(growthforecast|munin|zabbix)
out_(mail|mongo|redis|file|mysql|s3)
Mail /Store
out_copy out_exec_filter
out_sampling_filter
Recipe
fluent-agent-lite
rewrite
file-alternative
parser
forest
numeric_counter
fluent-plugin
growthforecast
最終章
開眼
Recipe
fluent-agent-lite
rewrite
file-alternative
parser
forest
numeric_counter
fluent-plugin
growthforecast
td-agent
in_forward
out_file_alternative
error log
access log
error log
access logfluent-agent-lite
access log
nginx
fluent-agent-lite
error log
WEB SERVERS
LOG SERVERMONITORINGSERVER
out_copy
out_parser
out_exec_filter
out_numeric_counter
out_(growthforecast|munin|zabbix)
out_(mail|mongo|redis|file|mysql|s3)
Mail /Store
out_copy out_exec_filter
td-agent
in_forward
out_file_alternative
error log
access log
error log
access logfluent-agent-lite
access log
nginx
fluent-agent-lite
error log
out_(growthforecast|munin|zabbix)
WEB SERVERS
LOG SERVERMONITORINGSERVER
out_copy
out_parser
out_exec_filter
out_rewrite
out_numeric_counter
out_(mail|mongo|redis|file|mysql|s3)
Mail /Store
out_copy out_exec_filter
Before rewrite
nginx.access:{"method":"GET","path":"/mypage/status","status":"200","response_time":"161576"}
Before rewrite
nginx.access:{"method":"GET","path":"/mypage/status","status":"200","response_time":"161576"}
After rewrite
nginx.access.mypage:{"method":"GET","path":"/mypage/status","status":"200","response_time":"161576"}
Point of configuration
<rule> key path pattern ^\/([^\/]+) fallback others append_to_tag true</rule>
Follow all request path
pattern!!1
Point of configuration
<rule> key path pattern ^\/([^\/]+) fallback others append_to_tag true</rule>
Follow all request path
pattern!!1
If the next step out_datacounter,
Be aware that there is an upper limit to the number of patterns!!1
1 #!/usr/bin/env perl$ 2 $ 3 use strict;$ 4 use warnings;$ 5 use Data::Dumper;$ 6 use DBIx::Handler;$ 7 use JSON::XS;$ 8 use Data::MessagePack;$ 9 $ 10 $| = 1;$ 11 $ 12 my $mp = Data::MessagePack->new();$ 13 $ 14 my $db_user = "root";$ 15 my $db_passwd = "";$ 16 my $db_opt = { RaiseError => 0, PrintError => 1 };$ 17 my $handler = DBIx::Handler->new( "DBI:mysql:database=information_schema:1 27.0.0.1", $db_user, $db_passwd, $db_opt );$ 18 $ 19 my $db;$ 20 my $long_query_time = $handler->dbh->selectrow_array($ 21 "SELECT VARIABLE_VALUE FROM GLOBAL_VARIABLES WHERE VARIABLE_NAME = 'LONG_Q UERY_TIME'");$ 22 $handler->dbh->do("SELECT SLEEP($long_query_time)");$ 23 $ 24 while ( my $json = <STDIN> ) {$ 25 my $slowlog = eval { decode_json($json); };$ 26 next if ($@);$
MySQL slow_log
MongoDB
27 $ 28 unless ( defined $slowlog->{sql} ) {$ 29 print $mp->pack($slowlog);$ 30 next;$ 31 }$ 32 $ 33 my $select_statement;$ 34 if ( $slowlog->{sql} =~ /(select[^\;]+)/i ) {$ 35 $select_statement = $1;$ 36 }$ 37 $ 38 if ( $slowlog->{sql} =~ /^use ([^\;]+)/i ) {$ 39 $db = '`' . $1 . '`';$ 40 }$ 41 $ 42 if ( defined $db && defined $select_statement ) {$ 43 $handler->dbh->do("use $db");$ 44 my $explains = eval { $handler->dbh->selectall_arrayref( "EXPLAIN $selec t_statement", +{ Slice => {} } ); };$ 45 if ( ref $explains eq 'ARRAY' ) {$ 46 $slowlog->{explain} = $explains;$ 47 $slowlog->{database} = $db;$ 48 }$ 49 }$ 50 $ 51 print $mp->pack($slowlog);$ 52 }$
tail
EXPLAIN
store
番外編
SLOW QUERY
1 #!/usr/bin/env perl$ 2 $ 3 use strict;$ 4 use warnings;$ 5 use Data::Dumper;$ 6 use DBIx::Handler;$ 7 use JSON::XS;$ 8 use Data::MessagePack;$ 9 $ 10 $| = 1;$ 11 $ 12 my $mp = Data::MessagePack->new();$ 13 $ 14 my $db_user = "root";$ 15 my $db_passwd = "";$ 16 my $db_opt = { RaiseError => 0, PrintError => 1 };$ 17 my $handler = DBIx::Handler->new( "DBI:mysql:database=information_schema:1 27.0.0.1", $db_user, $db_passwd, $db_opt );$ 18 $ 19 my $db;$ 20 my $long_query_time = $handler->dbh->selectrow_array($ 21 "SELECT VARIABLE_VALUE FROM GLOBAL_VARIABLES WHERE VARIABLE_NAME = 'LONG_Q UERY_TIME'");$ 22 $handler->dbh->do("SELECT SLEEP($long_query_time)");$ 23 $ 24 while ( my $json = <STDIN> ) {$ 25 my $slowlog = eval { decode_json($json); };$ 26 next if ($@);$
MySQL slow_log
MongoDB
27 $ 28 unless ( defined $slowlog->{sql} ) {$ 29 print $mp->pack($slowlog);$ 30 next;$ 31 }$ 32 $ 33 my $select_statement;$ 34 if ( $slowlog->{sql} =~ /(select[^\;]+)/i ) {$ 35 $select_statement = $1;$ 36 }$ 37 $ 38 if ( $slowlog->{sql} =~ /^use ([^\;]+)/i ) {$ 39 $db = '`' . $1 . '`';$ 40 }$ 41 $ 42 if ( defined $db && defined $select_statement ) {$ 43 $handler->dbh->do("use $db");$ 44 my $explains = eval { $handler->dbh->selectall_arrayref( "EXPLAIN $selec t_statement", +{ Slice => {} } ); };$ 45 if ( ref $explains eq 'ARRAY' ) {$ 46 $slowlog->{explain} = $explains;$ 47 $slowlog->{database} = $db;$ 48 }$ 49 }$ 50 $ 51 print $mp->pack($slowlog);$ 52 }$
tail
EXPLAIN
store
公開したよ！https://github.com/studio3104/out-exec-mysql-slowquery
スーパー宣伝タイムエピローグ
Openess is our driver for
excellence
Small Startup Fluentd
ご清聴ありがとうございました
@studio3104
Talk 4th Section at Fluentd Meetup #3
