2. 2 lApache Flume lFlume lApache FlumeRelease 3. 3 () Ameba
Technology Laboratory Hadoop/Hive AmebaFlume 4. Apache Flume 5. 5 l
l Cloudera1CDH l Source(), Channel(), Sink()3 l GoogleFlumeJava l
Doc : https://ume.apache.org Git : https://github.com/apache/ume
Ver. 1.5.1 Apache Flume 6. 6 l / l File Channel l HDFS, HBase l
Component(Sink, Intereptor) l Avro-RPC Flume 7. Flume in Ameba 8. 8
Hadoop Ameba Ameba Ameba Aggre gator Flume Aggregator Aggre gator
HBase (Onix) Recommend (HBase sink) Elastic Search 9. 9 Ameba l
1,600 host l 140,000 lines / sec l 2.0 TB / dayRaw l 80Mbps
(Compressed) 10. 10 Flume Agent (JVM) (1) l Source, Channel, Sink l
Channel Selector l HeaderChannel l Multiplexing l Optional channels
l Channelput l Channel Source Selector ChannelChannel Channel
SinkSink Sink Flume AgentHadoop HBase 11. 11 (2) l Failover l Sink
l Sinkpriority l LoadBalancing l Sink l Round_robin random l
Blacklist (backo) Flume Agent (JVM) Channel Sink Sink Source Flume
Agent (JVM) Channel Sink Source SinkSinkSink 12. 12 l HDFSHBase l
Cloudera Tokyo 2013
http://www.slideshare.net/cyberagent/cloudera-world-tokyo-2013 l
Hadoop Conference 2013
http://www.slideshare.net/iijiji0314/umeameba l SinkInterceptor 13.
13 Kibana l l Elastic Search 14. 14 l Flume ElasticSearch-Sink l
Serializer (JSON)KeyValueeld l Header (FLUME-2273 )
_index%{service_name}-yyyy-MM-dd _type %{activity_type} l l ES VM20
/ 8Core / Mem32G / HDD300G l 10,000 lines / sec l Index170G / day
(20) ElasticSearch 1.5.2VersionRelease Elastic Search Flume Agent
15. Apache Flume 16. 16 1.5 Released l spillable channel
(experimental) l Diskcapacity MemoryChannel l Kite dataset sink
(experimental) l Kite APIhdfsHBase l ElasticSearch HTTP API sink l
Much faster replay in the FileChannel Apache Flume (1) 17. 17 1.4
Released l Secure SSL transport over Avro l Avro l Thrift-RPC as a
transport mechanism l Embedding a Flume agent within apps l
Ingesting Avro-serializable objects via log4j API l JMS source l
ActiveMQ Apache Flume (2) 18. 18 l l 1 l HDFSAvroFlume1.5 l Flume12
http://shop.oreilly.com/product/0636920030348.do 19.
