1. Amazon Elastic MapReduceAWS Black Belt Tech Webinar 2014 ()2014/10/1

2. Agenda Amazon EMR Amazon EMR 3. Agenda Amazon EMR Amazon EMR 4. AWS Hadoop ServiceData Warehouse4Stream ProcessingAmazonKinesisCloud StorageAmazon S3NoSQLAmazon DynamoDBAmazonEMRAmazonRedshiftBI ToolsAmazon EC2RDBMSAmazon RDSData FlowAWS Data Pipeline 5. Hadoop-as-a-serviceWhatisEMR?Map-Reduce AWS 6. Amazon EMR 7. Amazon EMR 8. EMRAmazon EMR clusterJobTrackerNameNodeHivePigNode managementMaster nodeTaskTrackerDataNodeHDFSCore nodeCore instance groupTaskTrackerTask nodeAWS Cloud Master instance group Task instance group 9. EMR NameNode, JobTracker HadoopAmazon EMR clusterJobTrackerNameNodeHivePigNode managementMaster nodeAWS Cloud Master instance group 10. Amazon EMR TaskTrackerMapReduce DataNodeHDFS Master instance groupAmazon EMR clusterHDFS HDFSCore instance group 11. Amazon EMR HDFS CPURAM HDFSMaster instance groupAmazon EMR clusterHDFS HDFS HDFSCore instance group 12. Amazon EMR TaskTracker No HDFS Amazon S3HDFSMaster instance groupAmazon EMR clusterHDFS HDFSCore instance group Task instance group 13. Amazon EMR CPURAM Master instance groupAmazon EMR clusterHDFS HDFSCore instance group Task instance group 14. Agenda Amazon EMR Amazon EMR 15. EMR AMI 3.2.1 Amazon Linux 2014.03 Hadoop 2.4 Hive 0.13.1 Pig 0.12 HBase 0.94.18 Impala 1.2.4 Mahout 0.9 Cascading 2.5 Java: Oracle/Sun jdk-7u65 Perl 5.16.3 PHP 5.3.28 Python 2.6.9 R 3.0.2 Ruby 2.0 Scala 2.11.1 16. Hive 0.13 Support Hive121000 Hive12 Sub querySELECT * FROM AWHERE EXISTS (SELECT nullFROM BWHERE B.id = A.idAND B.date'2009-10-01')SELECT key FROM t1GROUP BY keyHAVING COUNT(value) IN (SELECT p FROM t2);Sub queryhttp://hortonworks.com/blog/announcing-apache-hive-0-13-completion-stinger-initiative/ 17. Kinesis Integration KinesisEMR Hive, Pig, MapReduce SQL EMR AMI 3.0.4~ DataSourcesDataSourcesDataSourcesRedShiftKinesisS3AppDynamoDBKinesisEMR 18. Kinesis Integration: Hive Kinesis StreamHive HQL JOIN Kinesis Stream S3, DynamoDB, HDFSHive table : Clickstream (Kinesis) JOIN Ad campaign data (DynamoDB)CREATE TABLE apache_logs (host STRING,...)STORED BY'com.amazon.emr.kinesis.hive.KinesisStorageHandler'TBLPROPERTIES(kinesis.stream.name=AccessLogStream);http://docs.aws.amazon.com/ElasticMapReduce/latest/DeveloperGuide/emr-kinesis.html 19. AWS CLI AWS CLIEMR$ aws emr create-cluster --name mycluster --ami-version 3.2.1 --instance-groups InstanceGroupType=MASTER,InstanceCount=1,InstanceType=m1.mediumInstanceGroupType=CORE,InstanceCount=2,InstanceType=m1.mediumaws emr add-steps --cluster-id j-xxxxxxxx --steps Type=STREAMING,Name='Word Count',ActionOnFailure=CONTINUE,Args=-files,s3://bucket/wordcount/scripts/wordcount-mapper.rb,-files,s3://bucket/wordcount/scripts/wordcount-reducer.rb,-mapper,wordcount-mapper.rb,-reducer,wordcount-reducer.rb,-input,s3://bucket/wordcount/input/,-output,s3://bucket/wordcount/output/http://docs.aws.amazon.com/cli/latest/reference/emr/index.html 20. EMRFS Consistent View Amazon S3Eventual Consistency LIST MapReduce EMRFS consistent view FS DynamoDBS3 EMRS3LIST retry ConsistencyException S3https://docs.aws.amazon.com/ElasticMapReduce/latest/DeveloperGuide/emr-plan-consistent-view.html 21. EMRFS Consistent View DynamoDB S3$ aws s3 ls s3://bucket/csv/2014-09-26 15:07:55 285325013 csv/ce53daa0-7e59-485c-858f-f09dd65ca14c-0000002014-09-26 15:07:37 285320703 csv/ce53daa0-7e59-485c-858f-f09dd65ca14c-0000012014-09-26 15:07:56 221967808 csv/ce53daa0-7e59-485c-858f-f09dd65ca14c-000002Objectsin S3Meta datainDynamoDB 22. EMRFS Consistent View Emrfs EMRFS$ emrfs --helpemrfs 1.0Usage: emrfs [describe-metadata|set-metadata-capacity|delete-metadata|create-metadata|list-metadata-stores|diff|delete|sync|import] [options] args... S3$ aws s3 rm s3://bucket/csv/528235e6-322a-4e29-bed3-ac937f713584-000000Directlydelete inhive select * from apache_logs_csv limit 10;S3OKFailed with exception java.io.IOException:com.amazon.ws.emr.hadoop.fs.consistency.exception.ConsistencyException: 1items inconsistent (no s3 object for associated metadata item). First object: /bucket/csv/528235e6-322a-4e29-bed3-ac937f713584-000000Time taken: 0.06 secondsConsistencyException 23. Agenda Amazon EMR Amazon EMR 24. EMR Amazon EMR https://media.amazonwebservices.com/jp/wp/AWS_Amazon_EMR_Best_Practices.pdf 25. Amazon EMR M3, C3, R3, I2, HS1 : M3C3CPU : xlarge 26. EMR Amazon S3 Data on Amazon S3 27. HDFSAmazon S3 Amazon S3 99.999999999% : Amazon S3 Amazon Glacier Amazon S3EMREMR 28. #1 Spot Spot Spot http://aws.amazon.com/ec2/purchasing-options/spot-instances/ 29. Spot HDFS MapReduce http://docs.aws.amazon.com/ElasticMapReduce/latest/DeveloperGuide/emr-plan-spot-instances.html 30. #2 Amazon S3HDFSHS148TBHDFSHS148TBHDFSAmazon S3 31. HS148TBHDFSHS148TBHDFSAmazon S3m3.xlm3.xlm3.xlm3.xlm3.xlm3.xlSpot Amazon S3 32. HS148TBHDFSHS148TBHDFSAmazon S3m3.xlm3.xlm3.xlm3.xlm3.xlm3.xl 33. 100MB Mapper MapperJVM JVM 34. HadoopAmazon S3 12GB mapperAmazon S3: 10MB~15MB/s Mapper6060sec * 15MB = 1GB 35. S3DistCp S3DistCphttp://docs.aws.amazon.com/ElasticMapReduce/latest/DeveloperGuide/UsingEMR_s3distcp.html 36. S3DistCp./elastic-mapreduce --jobflow j-3GY8JC4179IOK --jar /home/hadoop/lib/emr-s3distcp-1.0.jar --args '--src,s3://myawsbucket/cf,--dest,s3://myawsbucket/combined,--groupBy,.*XABCD12345678.([0-9]+-[0-9]+-[0-9]+-[0-9]+).*,--targetSize,1024,--outputCodec,lzo,--deleteOnSuccess 37. Agenda Amazon EMR Amazon EMR 38. Data managementAnalytics languagesHDFSAmazon EMRAmazonRedshiftAWS Data PipelineAmazonDynamoDBAmazon S3AmazonAmazon Kinesis RDS 39. Bootstrap Action Amazon EMR BashAmazon S3 https://github.com/awslabs/emr-bootstrap-actions 40. EMR as a Big data PlatformEMR MapReduce HBase: NoSQL Presto: SQL Impala:SQL Spark: 41. HBase on Amazon EMRhttp://docs.aws.amazon.com/ElasticMapReduce/latest/DeveloperGuide/emr-hbase.html 42. Apache HBase? HadoopNoSQL key value store (KVS) Hadoop HDFS MapReduce 43. HBase X 44. HBase on EMR Amazon S3 45. OpenTSDB HBase http://bit.ly/1wlJ0THhttp://opentsdb.net/ 46. Presto on Amazon EMR 47. Presto? SQL MapReduce GB PB Java 48. MapReduce vs. Prestohttp://treasure-data.hateblo.jp/entry/2014/09/04/133703 49. Presto HiveRDBMS MapReduce PrestoHive Hive PrestoANSI SQL S3 Redshift 50. Presto on EMR AMI 3.x http://bit.ly/1knbk19 EMRHDFSHive HBase() Cassandra Amazon S3 51. Impala on Amazon EMRhttp://docs.aws.amazon.com/ElasticMapReduce/latest/DeveloperGuide/emr-impala.html 52. Impala SQL MapReduce Presto C++ 53. Impala Hive Hive()Impala()Q1Q4http://docs.aws.amazon.com/ElasticMapReduce/latest/DeveloperGuide/impala-optimization.html 54. Impala HiveRDBMS MapReduce ImpalaHive HiveQLImpala 55. Impala on EMR AMI 3.x EMRHDFSHBase Amazon S3 HDFS ImpalaS3 56. Spark on Amazon EMRhttp://aws.amazon.com/articles/Elastic-MapReduce/4926593393724923 57. Apache Spark SQL HadoopAmazon S3Spark 58. Spark on EMR Spark 1.1bootstrap action http://bit.ly/1uZhF8wMasterNodeMaster instance groupHDFS HDFS32GB Memory 59. Spark on EMR S3 HDFSRDD RDD RDD S3 HDFSMasterNodeMaster instance groupHDFS HDFS32GB Memory 256GB MemoryInputOutput 60. Spark on EMR MasterNodeMaster instance groupHDFS HDFS32GB Memory 61. KinesisKinesis 62. EMR as a Big Data PlatformDaily/hourly batchHive / PigPresto /ImpalaDashboardBI ToolsHDFS/ S3HBase CanssaFast cluster processingInteractive queryndra MySQL KinesisSQL on any data setsStreamingSparkData analysis platform 63. Amazon EMRHadoop Hadoop Amazon EMR Amazon EMR MapReduce HBase, Presto, Impala, Spark 64. http://aws.amazon.com/documentation/elastic-mapreduce/ https://forums.aws.amazon.com/forum.jspa?forumID=52start=0 AWS Big Data Blog http://blogs.aws.amazon.com/bigdata 65. QA 66. Webinar AWS http://aws.amazon.com/jp/aws-jp-introduction/ 67. Appendix 68. Kinesis StreamEMR DynamoDB Data Pipeline / Crontab 69. HBase http://www.n10k.com/blog/hbase-for-architects/ 70. HBase : , : () http://blog.sematext.com/2012/07/16/hbase-memstore-what-you-should-know/ 71. PrestoClient CoordinatorWorkerWorkerWorkerConnectorPluginStorage / MetadataDiscoveryServicehttp://www.slideshare.net/frsyuki/presto-hadoop-conference-japan-2014 72. Impalahttp://www.theregister.co.uk/2012/10/24/cloudera_hadoop_impala_real_time_query/ 73. Spark Worker YARN, Mesos, Standalon Worker workerApp Executor Executorhttp://www.cs.berkeley.edu/~matei/papers/2012/nsdi_spark.pdfhttp://www.cs.berkeley.edu/~matei/papers/2010/hotcloud_spark.pdf 74. Spark () Resilient Distributed Datasets(RDDs) an efficient, general-purpose andfault-tolerant abstraction forsharing data in clusterapplications. In-memory Partitioned Immutable RDDs support transformationsand actions