Hadoop operation chaper 4

1. Copyright CELLNT Corp. All right Reserved. h t t p : / / w w w . x d a t a . j p / 4 Hadoop Hadoop 201529 R&D @sudabon 2. Copyright CELLNT Corp. All right Reserved. h t t p : / / w w w . x d a t a . j p / 4.1 Hadoop 2 3. Copyright CELLNT Corp. All right Reserved. h t t p : / / w w w . x d a t a . j p / @ Apache Software Foundation CDH @ Cloudera HDP @ HortonWorks CDH HDPCMAmbari MapR M7 @ MapR Technologies OSS HDFSLinux FS EMR @ AWS Azure @ Microsoft 4.1 Hadoop 3 4. Copyright CELLNT Corp. All right Reserved. h t t p : / / w w w . x d a t a . j p / CDH 5.3.0 avro bigtop cdk crunch : MR Pipeline datafu : UDF for Pig ume hadoop hbase hive hue impala kite : For Developers llama 4.1.2 Hadoop 4 1 mahout oozie parquet (parquet-format) pig search (solr) setnry spark sqoop sqoop2 : also MongoDB whirr : scripts for Cloud zookeeper 5. Copyright CELLNT Corp. All right Reserved. h t t p : / / w w w . x d a t a . j p / 1 2 1 0.20.0MRv1 2 0.23.0YARN(MRv2) 2.6 CDH2.5 4.1.3 Hadoop 5 6. Copyright CELLNT Corp. All right Reserved. h t t p : / / w w w . x d a t a . j p / RAIDJBOD(Just a Bunch Of Disks) 2 NIC RAID 1+0 RAM OSHadoop 4.1.57 6 7. Copyright CELLNT Corp. All right Reserved. h t t p : / / w w w . x d a t a . j p / 20... 42.6GHz CPU 2 24GB DDR3 RAM 1Gbps NIC SATA II 2 2 NameNode NameNode 1001GB 4.1.57 7 8. Copyright CELLNT Corp. All right Reserved. h t t p : / / w w w . x d a t a . j p / JobTracker 100 2030% CPU 4.1.57 8 9. Copyright CELLNT Corp. All right Reserved. h t t p : / / w w w . x d a t a . j p / 9 10. Copyright CELLNT Corp. All right Reserved. h t t p : / / w w w . x d a t a . j p / 10 11. Copyright CELLNT Corp. All right Reserved. h t t p : / / w w w . x d a t a . j p / map reduce reducer9 4.1.8 11 12. Copyright CELLNT Corp. All right Reserved. h t t p : / / w w w . x d a t a . j p / I/O SANNASHadoopI/O Hadoop I/O 4.1.9 SAN 12 13. Copyright CELLNT Corp. All right Reserved. h t t p : / / w w w . x d a t a . j p / 4.2 13 14. Copyright CELLNT Corp. All right Reserved. h t t p : / / w w w . x d a t a . j p / Linux Windows2 Hadoop NameNode100GB DataNode MR Hadoop Hadooppid HadooptempJAR JAVA 64bit Oracle JDK 4.2.12 OSJAVA 14 15. Copyright CELLNT Corp. All right Reserved. h t t p : / / w w w . x d a t a . j p / Hadoop DNIPNN NNIP DN DN InetAddress.getLocalHost() InetAddress#getCanonicalHostName() OS/Java DN NN 4.2.3 DNS 15 16. Copyright CELLNT Corp. All right Reserved. h t t p : / / w w w . x d a t a . j p / Hadoop HDFS hdfs MRv1 mapredMRv2 yarn LinuxPluggable Authentication Modules(PAM) hdfsmapred32k dfs.name.dir fs.checkpoint.dir dfs.data.dir mapred.local.dir $HADOOP_LOG_DIR hadoop.tmp.dir 4.2.4 16 17. Copyright CELLNT Corp. All right Reserved. h t t p : / / w w w . x d a t a . j p / vm.swappiness = 0 0100 ex) HBaseZookeeperHRS vm.overcommit_memory = 1 2 RAM 0 or 1 or 2 Hadoop Streaming Javavfork()fork() vm.overcommit_raito:50 1GB1.5GB 4.3 17 18. Copyright CELLNT Corp. All right Reserved. h t t p : / / w w w . x d a t a . j p / MRI/O ext3 or ext4 or xfs Logical Volume Manager Volume Group Volume Group Physical Volume1 ext3 4KB 2TB16TBFS ordered -m14% sparse_super: dir_index: B-tree 4.4 18 19. Copyright CELLNT Corp. All right Reserved. h t t p : / / w w w . x d a t a . j p / ext4 ext3 extent: I/O3 xfs ext3ext4 ext4extent ext3ext44 inode 4.4 19 20. Copyright CELLNT Corp. All right Reserved. h t t p : / / w w w . x d a t a . j p / noatime/etc/fstab noatime nodiratime noatimeOKnodiratime noatime read()5 4.4.2 20 21. Copyright CELLNT Corp. All right Reserved. h t t p : / / w w w . x d a t a . j p / HDFS S/C 4.5.1.1 21 22. Copyright CELLNT Corp. All right Reserved. h t t p : / / w w w . x d a t a . j p / MR 4.5.1.2 22 23. Copyright CELLNT Corp. All right Reserved. h t t p : / / w w w . x d a t a . j p / 1Gb vs 10Gb ETLExtract/Transform/Load 10Gb 1Gb HBase10Gb 1Gb10Gb 4.5.2 23 24. Copyright CELLNT Corp. All right Reserved. h t t p : / / w w w . x d a t a . j p / 576(12 x 48) 48Gb > 4 x 10Gb (=1.2:1) 24 4 48 25. Copyright CELLNT Corp. All right Reserved. h t t p : / / w w w . x d a t a . j p / 1152(24 x 48) 1.2:1 4.5.3.1 25 3 48 12 26. Copyright CELLNT Corp. All right Reserved. h t t p : / / w w w . x d a t a . j p / ETL 4.5.3.1 26 27. Copyright CELLNT Corp. All right Reserved. h t t p : / / w w w . x d a t a . j p / 1.2:1 27 L3 48 28. Copyright CELLNT Corp. All right Reserved. h t t p : / / w w w . x d a t a . j p / 1.2:1 2304 4.5.3.2 28 48 10GbE 10GbE 10GbE 10GbE (48) (2304) L3 1 1 1 1 29. Copyright CELLNT Corp. All right Reserved. h t t p : / / w w w . x d a t a . j p / static routing L3 4.5 29 30. Copyright CELLNT Corp. All right Reserved. h t t p : / / w w w . x d a t a . j p / 1: http://archive.cloudera.com/cdh5/cdh/5/ 2: http://qiita.com/moris/items/0a23bf26abc4289bb258 3: http://wiki.openwrt.org/doc/howto/storage 4: http://ja.wikipedia.org/wiki/XFS 5: http://shiumachi.hatenablog.com/entry/20080614/1213415948 30