17
Hadoop Edit by Cassell Hsu 2013.04.19

Ntb0e5xqqjihzzxh3bit 140416002604-phpapp02-140416020154-phpapp02

Embed Size (px)

Citation preview

HadoopEdit by Cassell Hsu

2013.04.19

Hadoop

Master

Slave

Slave

Slave

Hadoop•Master•NameNode•JobTracker•SecondaryNameNode

NameNode•Where is NameNode?•Master•HDFS(Hadoop Distributed File

System)•What is NameNode?•資料之位置資訊 •資料之屬性

??

NameNode

•位置資訊 ?•所有資料皆存放在 – DataNode

DataNode•What is DataNode•存放資料

•Where is DataNode•HDFS•Slaves (and Master)

UserUser

DataNodeDataNode

NameNodeNameNode

DataNodeDataNode

DataNodeDataNode

128Mb

B64Mb

A64Mb

Check hdfs-site.xml

B64Mb

A64Mb

Hadoop

DataNodeDataNodeNameNodeNameNode

MasterDataNodeDataNode

DataNodeDataNode

DataNodeDataNode

Slaves

Hadoop•Master•NameNode•JobTracker•SecondaryNameNode

JobTracker•What is JobTracker?•排程工作

•Where is JobTracker?•Master

誰來工作?

JobTracker & TaskTrackerJobTracker TaskTracker

Where Master Slaves

What 排程工作 執行工作

Hadoop

DataNodeDataNodeNameNodeNameNode

MasterDataNodeDataNode

Slaves

JobTrackerJobTracker

TaskTrackerTaskTracker

Hadoop•Master•NameNode•JobTracker•SecondaryNameNode

SecondaryNameNode

•What is SecondaryNameNode?•NameNode發生錯誤時補救

•Where is SecondaryNameNode?•Master

Hadoop

DataNodeDataNodeNameNodeNameNode

MasterDataNodeDataNode

Slaves

JobTrackerJobTracker

TaskTrackerTaskTracker

SecondaryNodeSecondaryNode

MapReduce

User Master

Slave2

Slave1

A

A1

A2Task

NameNode

Task

Result1Result2

ReduceFinal

ResultHDFS

MapReduce

•檔案切割•Hadoop上區塊切割•程式指定