HAdoop presentation

HadoopEdit by Cassell Hsu

2013.04.19

Hadoop

Master

Slave

Slave

Slave

Hadoop

• Master

• NameNode

• JobTracker

• SecondaryNameNode

NameNode• Where is NameNode?

• Master

• HDFS(Hadoop Distributed File System)

• What is NameNode?

• 資料之位置資訊

• 資料之屬性

？？

NameNode

• 位置資訊 ?

• 所有資料皆存放在 – DataNode

DataNode• What is DataNode

• 存放資料• Where is DataNode

• HDFS

• Slaves (and Master)

UserUser

DataNodeDataNode

NameNodeNameNode

DataNodeDataNode

DataNodeDataNode

128Mb B

64Mb

A64Mb

Check hdfs-site.xml

B64Mb

A64Mb

Hadoop

DataNodeDataNodeNameNodeNameNode

MasterDataNodeDataNode

DataNodeDataNode

DataNodeDataNode

Slaves

Hadoop

• Master

• NameNode

• JobTracker

• SecondaryNameNode

JobTracker

• What is JobTracker?

• 排程工作• Where is JobTracker?

• Master

誰來工作？

JobTracker & TaskTrackerJobTracker TaskTracker

Where Master Slaves

What 排程工作 執行工作

Hadoop

DataNodeDataNodeNameNodeNameNode

MasterDataNodeDataNode

Slaves

JobTrackerJobTracker

TaskTrackerTaskTracker

Hadoop

• Master

• NameNode

• JobTracker

• SecondaryNameNode

SecondaryNameNode

• What is SecondaryNameNode?

• NameNode發生錯誤時補救

• Where is SecondaryNameNode?

• Master

Hadoop

DataNodeDataNodeNameNodeNameNode

MasterDataNodeDataNode

Slaves

JobTrackerJobTracker

TaskTrackerTaskTracker

SecondaryNodeSecondaryNode

MapReduce

User Master

Slave2

Slave1

A

A1

A2Task

NameNode

Task

Result1

Result2

Reduce

Final ResultHDFS

MapReduce

• 案切割檔

• Hadoop上區塊切割

• 程式指定