= Hadoop分佈式文件系統使用指南二 = = 升級 = * 由於換版本的話,資料夾內的conf設定檔也勢必被更改,因此目前作法為: 把conf 移至/opt/conf ,hadoop 0.16 與 hadoop 0.18用 ln 做捷徑代換。由於conf已不在hadoop_home內,因此記得匯入conf/hadoop-env.sh {{{ $ source /opt/conf/hadoop-env.sh }}} * 先看狀態 {{{ $ bin/hadoop dfsadmin -upgradeProgress status There are no upgrades in progress. }}} * 停止hdfs * 注意不可使用bin/stop-all.sh來停止 {{{ $ bin/stop-dfs.sh }}} * 部署新版本的Hadoop * 注意每個node的版本都要統一,否則會出現問題 * 啟動 {{{ $ bin/start-dfs.sh -upgrade }}} * namenode管理網頁會出現升級狀態 = 退回 = * 停止集群 {{{ $ bin/stop-dfs.sh }}} * 部署老版本的Hadoop * 退回之前版本 {{{ $ bin/start-dfs.sh -rollback }}} = Hadoop指令手冊 = {{{ $ hadoop [--config confdir] [COMMAND] [GENERIC_OPTIONS] [COMMAND_OPTIONS] }}} [GENERIC_OPTIONS] : || -conf || 指定應用程序的配置文件。 || || -D || 為指定property指定值value。 || || -fs || 指定namenode。 || || -jt || 指定job tracker。只適用於job。 || == archieve == * archieve就是把資料壓縮成一個檔案,在壓縮的過程中,還會將被壓縮的目錄結構紀錄在index與masterindex內。 * 由於每個上傳上去的檔案都被放在一個block中,因此我的input資料夾內共有四個檔,但是每個檔都會佔用一個block,用此方法就可以按照整個打包大小來分配共用去多少個block數。 * hadoop archive -archiveName name * {{{ $ bin/hadoop archive -archiveName foo.har input/* output 09/04/02 14:02:30 WARN mapred.JobClient: Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same. 09/04/02 14:02:30 INFO mapred.JobClient: Running job: job_200904021140_0001 09/04/02 14:02:31 INFO mapred.JobClient: map 0% reduce 0% 09/04/02 14:02:44 INFO mapred.JobClient: map 20% reduce 0% 09/04/02 14:02:49 INFO mapred.JobClient: map 100% reduce 0% 09/04/02 14:02:56 INFO mapred.JobClient: Job complete: job_200904021140_0001 ...略 }}} * 看har裡面的檔案結構 {{{ $ bin/hadoop dfs -lsr /user/waue/output/foo.har }}} * 看har內檔案的內容 {{{ $ bin/hadoop dfs -cat /user/waue/output/foo.har/part-0 }}} * ps: 官方文件介紹的 hadoop dfs -lsr har:///user/hadoop/output/foo.har 會出現錯誤! {{{ #!sh lsr: could not get get listing for 'har:/user/waue/output/foo.har/user/waue' : File: har://hdfs-gm1.nchc.org.tw:9000/user/waue/output/foo.har/user/waue/input does not exist in har:///user/waue/output/foo.har }}} == distCp == * 是用於大規模集群內部和集群之間拷貝的工具 * 使用Map/Reduce實現文件分發,錯誤處理和恢復,以及報告生成 * 舉例為: {{{ hadoop distcp hdfs://nn1:8020/foo/bar hdfs://nn2:8020/bar/foo }}} ?? 然而8020 port 在機器上沒有開,且不是應該檔案會均勻散佈在每個節點上嗎?怎麼還會知道nn1的節點上有這個檔要複製到nn2呢? * 參考:http://cn.hadoop.org/doc/distcp.html == fsck == * HDFS文件系統檢查工具 {{{ $ bin/hadoop fsck / . /user/waue/input/1.txt: Under replicated blk_-90085106852013388_1001. Target Replicas is 3 but found 2 replica(s). /user/waue/input/1.txt: Under replicated blk_-4027196261436469955_1001. Target Replicas is 3 but found 2 replica(s). . /user/waue/input/2.txt: Under replicated blk_-2300843106107816641_1002. Target Replicas is 3 but found 2 replica(s). . /user/waue/input/3.txt: Under replicated blk_-1561577350198661966_1003. Target Replicas is 3 but found 2 replica(s). . /user/waue/input/4.txt: Under replicated blk_1316726598778579026_1004. Target Replicas is 3 but found 2 replica(s). Status: HEALTHY Total size: 143451003 B Total dirs: 8 Total files: 4 Total blocks (validated): 5 (avg. block size 28690200 B) Minimally replicated blocks: 5 (100.0 %) Over-replicated blocks: 0 (0.0 %) Under-replicated blocks: 5 (100.0 %) Mis-replicated blocks: 0 (0.0 %) Default replication factor: 3 Average block replication: 2.0 Corrupt blocks: 0 Missing replicas: 5 (50.0 %) Number of data-nodes: 2 Number of racks: 1 The filesystem under path '/' is HEALTHY }}} * 加不同的參數有不同的用處,如 {{{ $ bin/hadoop fsck / -files /tmp /tmp/hadoop /tmp/hadoop/hadoop-waue /tmp/hadoop/hadoop-waue/mapred /tmp/hadoop/hadoop-waue/mapred/system /user /user/waue /user/waue/input /user/waue/input/1.txt 115045564 bytes, 2 block(s): Under replicated blk_-90085106852013388_1001. Target Replicas is 3 but found 2 replica(s). Under replicated blk_-4027196261436469955_1001. Target Replicas is 3 but found 2 replica(s). /user/waue/input/2.txt 987864 bytes, 1 block(s): Under replicated blk_-2300843106107816641_1002. Target Replicas is 3 but found 2 replica(s). /user/waue/input/3.txt 1573048 bytes, 1 block(s): Under replicated blk_-1561577350198661966_1003. Target Replicas is 3 but found 2 replica(s). /user/waue/input/4.txt 25844527 bytes, 1 block(s): Under replicated blk_1316726598778579026_1004. Target Replicas is 3 but found 2 replica(s). Status: HEALTHY ....(同上) }}} || -move || -delete || -openforwrite || || 移動受損文件到/lost+found || 刪除受損文件 || 印出寫打開的文件 || || -files || -blocks || -locations || -racks || || 印出正被檢查的文件 || 印出區塊的資訊 || 印出每個區塊的位置 || 印出data-node的網絡拓撲結構 || 如: {{{ $ bin/hadoop fsck /user/waue/input/1.txt -files -blocks -locations /user/waue/input/1.txt 115045564 bytes, 2 block(s): Under replicated blk_-90085106852013388_1001. Target Replicas is 3 but found 2 replica(s). Under replicated blk_-4027196261436469955_1001. Target Replicas is 3 but found 2 replica(s). 0. blk_-90085106852013388_1001 len=67108864 repl=2 [140.110.138.191:50010, 140.110.141.129:50010] 1. blk_-4027196261436469955_1001 len=47936700 repl=2 [140.110.138.191:50010, 140.110.141.129:50010] Status: HEALTHY Total size: 115045564 B Total dirs: 0 Total files: 1 ....(略) }}}