17
Hadoop Edit by Cassell Hsu 2013.04.19

Test schedule

Embed Size (px)

Citation preview

Page 1: Test schedule

HadoopEdit by Cassell Hsu

2013.04.19

Page 2: Test schedule

Hadoop

Master

Slave

Slave

Slave

Page 3: Test schedule

Hadoop

•Master

•NameNode

•JobTracker

•SecondaryNameNode

Page 4: Test schedule

NameNode•Where is NameNode?

•Master

•HDFS(Hadoop Distributed File System)

•What is NameNode?

•資料之位置資訊

•資料之屬性

??

Page 5: Test schedule

NameNode

•位置資訊 ?

•所有資料皆存放在 – DataNode

Page 6: Test schedule

DataNode

•What is DataNode

•存放資料•Where is DataNode

•HDFS

•Slaves (and Master)

Page 7: Test schedule

UserUser

DataNodeDataNode

NameNodeNameNode

DataNodeDataNode

DataNodeDataNode

128Mb

B64Mb

A64Mb

Check hdfs-site.xml

B64Mb

A64Mb

Page 8: Test schedule

Hadoop

DataNodeDataNodeNameNodeNameNode

MasterDataNodeDataNode

DataNodeDataNode

DataNodeDataNode

Slaves

Page 9: Test schedule

Hadoop

•Master

•NameNode

•JobTracker

•SecondaryNameNode

Page 10: Test schedule

JobTracker

•What is JobTracker?

•排程工作•Where is JobTracker?

•Master

誰來工作?

Page 11: Test schedule

JobTracker & TaskTrackerJobTracker TaskTracker

Where Master Slaves

What 排程工作 執行工作

Page 12: Test schedule

Hadoop

DataNodeDataNodeNameNodeNameNode

MasterDataNodeDataNode

Slaves

JobTrackerJobTracker

TaskTrackerTaskTracker

Page 13: Test schedule

Hadoop

•Master

•NameNode

•JobTracker

•SecondaryNameNode

Page 14: Test schedule

SecondaryNameNode

•What is SecondaryNameNode?

•NameNode發生錯誤時補救

•Where is SecondaryNameNode?

•Master

Page 15: Test schedule

Hadoop

DataNodeDataNodeNameNodeNameNode

MasterDataNodeDataNode

Slaves

JobTrackerJobTracker

TaskTrackerTaskTracker

SecondaryNodeSecondaryNode

Page 16: Test schedule

MapReduce

User Master

Slave2

Slave1

A

A1

A2Task

NameNode

Task

Result1

Result2

Reduce

Final Result

HDFS

Page 17: Test schedule

MapReduce

•檔案切割•Hadoop上區塊切割

•程式指定