View
22
Download
2
Category
Preview:
Citation preview
Been Stuck In Elastic StackKazuhiro Kosaka
Engineer, MDH, CyberAgent, Inc.
Feb 10, 2017
About Me• 小坂和弘 - Kazuhiro Kosaka @hyperdash• 2009- CyberAgent, Inc.• Designer/Front-end Engineer/Back-end Engineer• Motion Graphics, VJ
About Me• I love hiking
Summary
What To Talk?• Amazon Elasticsearch Service• v1.x v2.x
Summary
What NOT To Talk?• Private Elasticsearch Cluster / Elastic Cloud• v5.x
Summary
Elastic Stack?• Had been called ELK Stack
• Elasticsearch - Distributed Search/Analytics Engine• Logstash - Data Processing Pipeline• Kibana - Data Visualizer• Beats - Data Shipping Agents
Our Case広告運用・配信設定管理画面 ユーザ操作ログ
Our Case• Elasticsearch v1.5 on Amazon Elasticsearch Service• Logstash v2.3.1• Filebeat v1.2.1• m3.large / 3 instances• 400,000 docs / day
• 95% of logs are by 日予算自動アロケーション Batch• Will be separated into another index
• 主にユーザの操作履歴調査
Our Case
Stuck #1 / SettingsDataLoss を防ぐには？
• Filebeat uses a back-pressure sensitive protocol• Logstash < v5.1 has NO Persistent Queue
Stuck #1 / SettingsDataLoss を防ぐには？
Source: https://www.elastic.co/guide/en/logstash/5.0/deploying-and-scaling.html
Stuck #1 / SettingsDataLoss を防ぐには？
• Use Message Queue e.g. Kafka• Delete Filebeat registrar file and Re-ship data
• >= v5.1 Persistent Queue (beta) is enabled
Stuck #2 / JVMMemoryPressureES に Write できなくなったんですけど…
• m3.medium / 1 instance• JVMMemoryPressure ≒ 92-94 %• FreeStorageSpace looks okay
Stuck #2 / JVMMemoryPressureES に Write できなくなったんですけど…
Source: http://docs.aws.amazon.com/elasticsearch-service/latest/developerguide/aes-handling-errors.html
• NOT a t2 instance• ???
Stuck #2 / JVMMemoryPressureES に Write できなくなったんですけど…
m3.medium / 1 instance
m3.large / 3 instances
Worked!
Stuck #2 / JVMMemoryPressureES に Write できなくなったんですけど…
• indices.fielddata.cache.size: unbounded (default)This allows ES to allocate unlimited memory for fielddata
• indices.fielddata.cache.size: 50 (%)
Stuck #2 / JVMMemoryPressureES に Write できなくなったんですけど…
Source: https://www.elastic.co/guide/en/elasticsearch/guide/current/_limiting_memory_usage.html
Stuck #2 / JVMMemoryPressureES に Write できなくなったんですけど…
• ES v1.x• Doc Values: Default OFF• Set “doc_values”: true in index template - mappings
Stuck #2 / JVMMemoryPressureES に Write できなくなったんですけど…
• ES >= v2.x• Doc Values: Default ON
Stuck #3 / RestrictionsAmazon Elasticsearch Service の制限
• Dynamic Scripting is not supported• Custom Plugins are not supported• Access Control
• Amazon ES is out of VPC• IP-Based, IAM-Based Access Control• logstash-output-amazon_es plugin is not stable??
Next Step
• Migrate to Elastic Stack v5.x• Elastic Cloud or Private Cluster?
• 管理画面上での操作ログ閲覧
Conclusion
• Amazon Elasticsearch Service
• Elastic Cloud• Easy & Handy (Fully-Managed)• The latest version is available anytime• Less Restrictions, or Elastic Cloud Enterprise (alpha)• Supported by Elastic
• Easy & Handy (Fully-Managed)• Restrictions• Supported by AWS
END
THANK YOU
Recommended