What could hadoop do for us

  • View
    330

  • Download
    7

Embed Size (px)

DESCRIPTION

What could Hadoop do for us ? HDFS / MapReduce

Text of What could hadoop do for us

  • Hadoop? - Simon Hsu 2014620
  • 2
  • 3
  • 4 About Me (Simon Hsu) 98 / 100 Hadoop () A Transparent Approach to Run MapReduce Programs on Collaborative Hadoops IEEE BigData 2014 - - Hadoop/ -Etu() Hadoop (Etu/Cloudera) /
  • 5 Overview Hadoop & Big Data Hadoop HDFS / MapReduce Workflow Hadoop Ecosystem Tools Introduction Resources
  • 6 Her () http://www.huffingtonpost.com/marshall-fine/movie- review-iheri_b_4459420.html
  • 7 We live in an age of rapid change.. http://b0.rimg.tw/ciltw/22fde0c3.jpg http://pic.pimg.tw/fyu45/1358000174-162486648.jpg (Before Steve jobs released iPhone in 2007..)
  • 8 We live in an age of rapid change.. 2007 http://www.computerhistory.org/atchm/steve-jobs/ http://www.businessinsider.com.au/yahoo-wants-to-keep-users-engaged-with-one-shop-shop-mobile-video-app-2013-9
  • 9 Jerrys Siri Screenshots
  • 10 Big Data? Etu https://www.youtube.com/watch?v=wc2durk8p9o
  • 11http://media2.hpcwire.com/datanami/hadoopelephant.jpg
  • 12 Transcedence () http://moviefloss.com/transcendence-movie-review- human-one-day-computer/
  • 13 Stephen Hawking: The creation of true AI could be the 'greatest event in human history http://www.independent.co.uk/news/science/stephen-hawking-transcendence-looks-at-the- implications-of-artificial-intelligence--but-are-we-taking-ai-seriously-enough-9313474.html
  • 14 says a group of leading scientists.. Success in creating AI would be the biggest event in human history. Unfortunately, it might also be the last, unless we learn how to avoid the risks ? ? http://www.independent.co.uk/news/science/stephen-hawking-transcendence-looks-at-the- implications-of-artificial-intelligence--but-are-we-taking-ai-seriously-enough-9313474.html
  • 15 Big Data 2014.6
  • 16 - IoT
  • 17 2014 6
  • 18 (1/2) 2014 6
  • 19 (2/2) 2014 6
  • 20 FamilyAsyst @Computex 2014
  • 21 FamilyAsyst Screenshots
  • 22 3Vs in Big Data http://www.geektime.com/2013/10/24/the-3-vs-of-big-data- and-their-technologies/
  • 23 Brief of Hadoop Hadoop Doug Cutting Hadoop (Locality) HDFS MapReduce http://www.cnbc.com/id/100769719
  • 24 Relations between Hadoop and Google The Google File System 2003 SOSP MapReduce : Simplified Data Processing on Large Cluster 2004 OSDI Bigtable : A Distributed Storage System for Structured Data 2006 OSDI Hadoop Distributed File System (Storage) MapReduce framework (Processing) HBase (Database) Hadoop Community
  • 25 HDFS NameNode (FileSystem Image) File Index (with meta data) Mapping of File and Block Locations of each Block (Journal) Operations of Namespace DataNode NameNode FileSystem Image Journal Logs of creating, deletion, rename of the namespace Root DirDir Dir File Block Block
  • 26 HDFS DataNode Block Block : 128MB NameNode BlockNameNode
  • 27 HDFS Local file system (/home/simon/testinput) DFS Shell / DFS API NN DN DN HDFS /user/simon/testinput HadoopHadoop
  • 28 Hadoop Data Distribution http://www.cloudera.com/content/cloudera/en/produ cts-and-services/cdh/hdfs-and-mapreduce.html Logical View Physical View
  • 29 HDFS NameNode (replication factor : 2) Agent File 1 DataNode 1 DataNode 2 DataNode 3 NameNodeFile1 ()1 Data Block Block 1 DN1DN2 Block 2 DN2DN3 3 File1Block ( Block 1 Block 2) 2 NameNodeAgent (BlockDN) () () () () 4 File 1 Block 1 Block 2 = Block 1 Block 2Block 2Block 1
  • 30 HDFS NameNode (replication factor : 2) Agent DataNode 1 DataNode 2 DataNode 3 NameNodeFile1 ( DFS Shell / DFS API )1 NameNode Block Block1DN1DN2 Block2DN2DN3 3 NameNode File1Block1Block22 NameNode () () () () () 4 File 1 Block 1 Block 2 = Block 1 Block 2Block 2Block 1
  • 31 Then, How MapReduce work? http://joyreactor.com/post/821302
  • 32 MapReduce JobTracker () JobTracker map reduce Locality Feedbacks of heartbeat (failure node / faster node) TaskTrackermap worker reduce worker TaskTracker () TaskTracker2worker (map worker or reduce worker) worker JobTracker map function reduce function
  • 33 MapReduce 1 ~ 300 301 ~ 600 601 ~ 900 map mapmap reduce
  • 34 http://www.slideshare.net/waue/hadoop- map-reduce-3019713 MapReduce Workflow
  • 35 WordCount Example Hi, be a winner, do not be a loser. map map map Hi, 1 be, 1 a, 1 winner, 1 do, 1 not, 1 be, 1 a, 1 loser, 1 reduce reduce a, 2 be, 2 do, 1 loser, 1 not, 1 winner, 1 Hi, 1 a, 2 be, 2 do, 1 loser, 1 not, 1 winner, 1 Hi, 1
  • 36 Hadoop Ecosystem (Still growing rapidly) Hadoop Distributed File System (File System) MapReduce (Processing) Sqoop / Flume (Data Integration) Pig / Hive (Analytical language) Mahout (Data mining) . HBase (Database) Zookeeper (Lock service)
  • 37 Example : MapReduce vs Hive Map/Reduce Hive
  • 38 Map/Reduce Pig Example : MapReduce vs Pig About 40% of M/R jobs in Yahoo are written using Pig
  • 39 Sqoop Hadoop Eco-System 1. RDBMS HDFS / Hbase / Hive 2. HDFS / Hbase / Hive RDBMS map-only job Sqoop http://blog.cloudera.com/blog/2012/01/apache -sqoop-highlights-of-sqoop-2/
  • 40 Hadoop at glance.. http://ambuj4bigdata.blogspot.tw/2014/05/hadoop-at-glance.html
  • 41 - (5X1) : Python http://www.etusolution.com/DSP/edm_dsp_ETL2.html http://goo.gl/gYNFW1 http://simonhsu.github.io/rent/
  • 42 About Etu Etu - Big Data Solution Etu Appliance - Big Data TQuery - Etu Recommender - Any
  • 43 - / Taiwan Hadoop User Group https://www.facebook.com/groups/hadoop.tw Hadoop Taiwan (workshop) http://www.hadoop.tw/ Hadoop Weekly (Mailing List) http://www.hadoopweekly.com/ Experfy (Big Data5945) https://www.experfy.com/ Top Coder http://www.topcoder.com/
  • 44 Resources from Etu Etu Taiwan (ex.) Hadoop Hadoop ( ) () http://www.etusolution.com/index.php/tw/product-and- services/etu-services/training-service EHC (Hadoop ) https://www.youtube.com/watch?v=OWVsmVu_PV8 DSP (Data Scientist Program) Etu CfT (Code for Tommorrow ) http://datasci.co/
  • 45 http://compbio.ucdenver.edu/Hunter_lab/Phang/resources/Harvar d_Data-Scientist-The-sexiest-job-of-the-21st-century_2012.pdf
  • 318, Rueiguang Rd., Taipei 114, Taiwan Simon Hsu Technical Manager 0912-166-961 simonhsu@etusolution.com Thank you