HadoopEdit by Cassell Hsu

2013.04.19

Hadoop

Master

Slave

Slave

Slave

Hadoop

•Master

•NameNode

•JobTracker

•SecondaryNameNode

NameNode•Where is NameNode?

•Master

•HDFS(Hadoop Distributed File System)

•What is NameNode?

•資料之位置資訊

•資料之屬性

？？

NameNode

•位置資訊 ?

•所有資料皆存放在 – DataNode

DataNode

•What is DataNode

•存放資料

•Where is DataNode

•HDFS

•Slaves (and Master)

User

DataNode

NameNode

DataNode

DataNode

128Mb

B64Mb

A64Mb

Check hdfs-site.xml

B64Mb

A64Mb

Hadoop

DataNodeNameNode

MasterDataNode

DataNode

DataNode

Slaves

Hadoop

•Master

•NameNode

•JobTracker

•SecondaryNameNode

JobTracker

•What is JobTracker?

•排程工作

•Where is JobTracker?

•Master

誰來工作？

JobTracker & TaskTrackerJobTracker TaskTracker

Where Master Slaves

What 排程工作 執行工作

Hadoop

DataNodeNameNode

MasterDataNode

Slaves

JobTracker

TaskTracker

Hadoop

•Master

•NameNode

•JobTracker

•SecondaryNameNode

SecondaryNameNode

•What is SecondaryNameNode?

•NameNode 發生錯誤時補救

•Where is SecondaryNameNode?

•Master

Hadoop

DataNodeNameNode

MasterDataNode

Slaves

JobTracker

TaskTracker

SecondaryNode

MapReduce

User Master

Slave2

Slave1

A

A1

A2Task

NameNode

Task

Result1

Result2

Reduce

Final Result

HDFS

MapReduce

•檔案切割

•Hadoop 上區塊切割

•程式指定