Gaining Operational Insights out of your logs

Kobi Biton, Solutions Architect

© 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

Agenda

• Common Log sources on AWS• Loading Streaming data into Amazon

Elasticsearch Service• Demo: Real world scenario

© 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

We want to turn this…

{"eventTime": "2016-06-06T09:59:55Z","eventSource": "signin.amazonaws.com","eventName": "ConsoleLogin","awsRegion": "us-east-1","sourceIPAddress": ”169.254.169.254","userAgent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.10; rv:38.0)

Gecko/20100101 Firefox/38.0",

"errorMessage": "Failed authentication","requestParameters": null,"responseElements": {

"ConsoleLogin": "Failure"},"additionalEventData": {

"LoginTo": "SomeInfo""MFAUsed": "Yes"

},{...}

© 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

Into this…

© 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

And then optionally this…

Amazon SNS

Topic

AWS Lambda

CloudwatchLogs

Alarms

© 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

Common Log Sources

• Amazon Cloudfront access logs , S3 server access logs

• Elastic Load Balancer access logs• AWS Cloudtrail , VPC Flow Logs• Amazon SNS notifications• Amazon EC2 OS Logs / Custom application

logs

© 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

Loading Streaming Data Into Amazon Elasticsearch Service

© 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

From Amazon S3

Amazon ESDomain

AWS Lambda

ELB Logs

Cloudfront Access Logs

S3 Access Logs

S3

© 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

AWS Cloudtrail

VPC Flow Logs

CloudwatchLogs

Amazon ESDomain

AWS Lambda

From Amazon Cloudwatch Logs

Cloudwatch Alarms

© 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

Kinesis FirehoseAgent

Amazon Kinesis

Firehose

Amazon ESDomain

Via Amazon Kinesis Firehose

© 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

LogstashAgent

Log Objects

Amazon ESDomain

Via Logstash Agent

© 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

DEMO

kobibito@amazon.com

© 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

Appendix

© 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

Useful Links:

• https://docs.aws.amazon.com/elasticsearch-service/latest/developerguide/es-aws-integrations.html

• https://github.com/awslabs/cloudwatch-logs-subscription-consumer/tree/master/configuration/kibana

• https://docs.aws.amazon.com/firehose/latest/dev/writing-with-agents.html• https://docs.aws.amazon.com/AmazonCloudWatch/latest/DeveloperGuide/Su

bscriptions.html• https://docs.aws.amazon.com/AmazonCloudWatch/latest/DeveloperGuide/M

onitoringLogData.html