é–¢è¥Hadoop‹‰¼·¼#1 Hadoop®ç´¹»‹

  • View
    1.663

  • Download
    4

Embed Size (px)

DESCRIPTION

第1回関西Hadoop勉強会でのプレゼンテーションです。入門編的な内容です。

Text of é–¢è¥Hadoop‹‰¼·¼#1...

MapReduce/ Hadoop

MapReduce/

Hadoopdragan10@gmail.comtamagawa_ryuji@twitterHadoop

HadoopSilverlightWebProgramming Google App EngineProgramming Windows Azure Windows Azurehttp://slidesha.re/9vo7fV

HadoopWeb1MySQL

2memcachedACID316128GBRAM15,000rpmMySQL4567RDBMS==I/OIn pioneer days they used oxen for heavy pulling, and when one ox couldnt budge a log, they didnt try to grow a larger ox. We shouldnt be trying for bigger computers, but for more systems of computers.Grace Hopper

CPUCPUSETI@HOMEFolding@HOMEHadoop

HadoopMapReduce/Apache FoundationMapReduce/HDFSAmazon EC2/S3Hadoop

HadoopMapReduceHDFSHadoopHDFSMapReduceMapReduceMapReduceHDFSHadoop Distributed File System64MBHadoop3AA-1DataBloockA-2DataBloockA-3DataBloockBB-1DataBloockDataBloockB-2DataBloockDataBloockB-3DataBloockDataBloockDataBloockDataBloockDataBloockDataBloockDataBloockDataBloockDataBloockDataBloockDataBloock0067011990999991950051507004...9999999N9+00001+99999999999...0043011990999991950051512004...9999999N9+00221+99999999999...0043011990999991950051518004...9999999N9-00111+99999999999...0043012650999991949032412004...0500001N9+01111+99999999999...0043012650999991949032418004...0500001N9+00781+99999999999...(0, 0067011990999991950051507004...9999999N9+00001+99999999999...)(106, 0043011990999991950051512004...9999999N9+00221+99999999999...)(212, 0043011990999991950051518004...9999999N9-00111+99999999999...)(318, 0043012650999991949032412004...0500001N9+01111+99999999999...)(424, 0043012650999991949032418004...0500001N9+00781+99999999999...)(1950, 0)(1950, 22)(1950, 11)(1949, 111)(1949, 78)(1949, [111, 78, 119, 76])(1950, [0, 22, 11, 3, 21, -5])(1949, 119)(1950, 22)HadoopmapHadoopreduce(1950, 3)(1950, 21)(1950, 5)(1949, 119)(1949, 76)mapfor line in sys.stdin: val = line.strip() (year, temp, q) = (val[15:19], val[87:92], val[92:93]) if (temp != "+9999" and re.match("[01459]", q)): print "%s\t%s" % (year, temp)(last_key, max_val) = (None, 0)for line in sys.stdin: (key, val) = line.strip().split("\t") if last_key and last_key != key: print "%s\t%s" % (last_key, max_val) (last_key, max_val) = (key, int(val)) else: (last_key, max_val) = (key, max(max_val, int(val)))if last_key: print "%s\t%s" % (last_key, max_val)Hadoop StreamingJavaHadoop Confefence Japan 2009NoSQL Not Only SQLNot Only RDBSQL / RDBNo SQLRDBBigDataRDBGoogle App EngineWindows AzureRDBFree lunchCassandraMongoDBTwitterClouderaPublickey http://www.publickey1.jp/index.html