Changes between Version 15 and Version 16 of waue/2009/0402


Ignore:
Timestamp:
Apr 2, 2009, 3:15:58 PM (17 years ago)
Author:
waue
Comment:

--

Legend:

Unmodified
Added
Removed
Modified

  • waue/2009/0402

    v15 v16  
    4444}}}
    4545
    46 = 檢查hdfs磁碟狀況 =
    47  * fsck 可以
    48 {{{
    49 $ bin/hadoop fsck /
    50 
    51 .
    52 /user/waue/input/1.txt:  Under replicated blk_-90085106852013388_1001. Target Replicas is 3 but found 2 replica(s).
    53 /user/waue/input/1.txt:  Under replicated blk_-4027196261436469955_1001. Target Replicas is 3 but found 2 replica(s).
    54 .
    55 /user/waue/input/2.txt:  Under replicated blk_-2300843106107816641_1002. Target Replicas is 3 but found 2 replica(s).
    56 .
    57 /user/waue/input/3.txt:  Under replicated blk_-1561577350198661966_1003. Target Replicas is 3 but found 2 replica(s).
    58 .
    59 /user/waue/input/4.txt:  Under replicated blk_1316726598778579026_1004. Target Replicas is 3 but found 2 replica(s).
    60 Status: HEALTHY
    61  Total size:    143451003 B
    62  Total dirs:    8
    63  Total files:   4
    64  Total blocks (validated):      5 (avg. block size 28690200 B)
    65  Minimally replicated blocks:   5 (100.0 %)
    66  Over-replicated blocks:        0 (0.0 %)
    67  Under-replicated blocks:       5 (100.0 %)
    68  Mis-replicated blocks:         0 (0.0 %)
    69  Default replication factor:    3
    70  Average block replication:     2.0
    71  Corrupt blocks:                0
    72  Missing replicas:              5 (50.0 %)
    73  Number of data-nodes:          2
    74  Number of racks:               1
    75 The filesystem under path '/' is HEALTHY
    76 }}}
    77 
    78  * 加不同的參數有不同的用處，如
    79 {{{
    80 $ bin/hadoop fsck / -files
    81 
    82 /tmp <dir>
    83 /tmp/hadoop <dir>
    84 /tmp/hadoop/hadoop-waue <dir>
    85 /tmp/hadoop/hadoop-waue/mapred <dir>
    86 /tmp/hadoop/hadoop-waue/mapred/system <dir>
    87 /user <dir>
    88 /user/waue <dir>
    89 /user/waue/input <dir>
    90 /user/waue/input/1.txt 115045564 bytes, 2 block(s):  Under replicated blk_-90085106852013388_1001. Target Replicas is 3 but found 2 replica(s).
    91  Under replicated blk_-4027196261436469955_1001. Target Replicas is 3 but found 2 replica(s).
    92 /user/waue/input/2.txt 987864 bytes, 1 block(s):  Under replicated blk_-2300843106107816641_1002. Target Replicas is 3 but found 2 replica(s).
    93 /user/waue/input/3.txt 1573048 bytes, 1 block(s):  Under replicated blk_-1561577350198661966_1003. Target Replicas is 3 but found 2 replica(s).
    94 /user/waue/input/4.txt 25844527 bytes, 1 block(s):  Under replicated blk_1316726598778579026_1004. Target Replicas is 3 but found 2 replica(s).
    95 Status: HEALTHY
    96 ....（同上）
    97 }}}
    9846
    9947= Hadoop指令手冊 =
    … …  
    15098？？ 然而8020 port 在機器上沒有開，且不是應該檔案會均勻散佈在每個節點上嗎？怎麼還會知道nn1的節點上有這個檔要複製到nn2呢？
    15199 * 參考：http://cn.hadoop.org/doc/distcp.html
     100
     101 == fsck ==
     102 * HDFS文件系統檢查工具
     103
     104{{{
     105$ bin/hadoop fsck /
     106
     107.
     108/user/waue/input/1.txt:  Under replicated blk_-90085106852013388_1001. Target Replicas is 3 but found 2 replica(s).
     109/user/waue/input/1.txt:  Under replicated blk_-4027196261436469955_1001. Target Replicas is 3 but found 2 replica(s).
     110.
     111/user/waue/input/2.txt:  Under replicated blk_-2300843106107816641_1002. Target Replicas is 3 but found 2 replica(s).
     112.
     113/user/waue/input/3.txt:  Under replicated blk_-1561577350198661966_1003. Target Replicas is 3 but found 2 replica(s).
     114.
     115/user/waue/input/4.txt:  Under replicated blk_1316726598778579026_1004. Target Replicas is 3 but found 2 replica(s).
     116Status: HEALTHY
     117 Total size:    143451003 B
     118 Total dirs:    8
     119 Total files:   4
     120 Total blocks (validated):      5 (avg. block size 28690200 B)
     121 Minimally replicated blocks:   5 (100.0 %)
     122 Over-replicated blocks:        0 (0.0 %)
     123 Under-replicated blocks:       5 (100.0 %)
     124 Mis-replicated blocks:         0 (0.0 %)
     125 Default replication factor:    3
     126 Average block replication:     2.0
     127 Corrupt blocks:                0
     128 Missing replicas:              5 (50.0 %)
     129 Number of data-nodes:          2
     130 Number of racks:               1
     131The filesystem under path '/' is HEALTHY
     132}}}
     133
     134 * 加不同的參數有不同的用處，如
     135{{{
     136$ bin/hadoop fsck / -files
     137
     138/tmp <dir>
     139/tmp/hadoop <dir>
     140/tmp/hadoop/hadoop-waue <dir>
     141/tmp/hadoop/hadoop-waue/mapred <dir>
     142/tmp/hadoop/hadoop-waue/mapred/system <dir>
     143/user <dir>
     144/user/waue <dir>
     145/user/waue/input <dir>
     146/user/waue/input/1.txt 115045564 bytes, 2 block(s):  Under replicated blk_-90085106852013388_1001. Target Replicas is 3 but found 2 replica(s).
     147 Under replicated blk_-4027196261436469955_1001. Target Replicas is 3 but found 2 replica(s).
     148/user/waue/input/2.txt 987864 bytes, 1 block(s):  Under replicated blk_-2300843106107816641_1002. Target Replicas is 3 but found 2 replica(s).
     149/user/waue/input/3.txt 1573048 bytes, 1 block(s):  Under replicated blk_-1561577350198661966_1003. Target Replicas is 3 but found 2 replica(s).
     150/user/waue/input/4.txt 25844527 bytes, 1 block(s):  Under replicated blk_1316726598778579026_1004. Target Replicas is 3 but found 2 replica(s).
     151Status: HEALTHY
     152....（同上）
     153}}}
     154
     155 === -move ===
     156 * 移動受損文件到/lost+found
     157 === -delete ===
     158 * 刪除受損文件。
     159 === -openforwrite ===
     160 * 印出寫打開的文件。
     161
     162 === -files ===
     163 * 印出正被檢查的文件
     164 === -blocks ===
     165 * 印出區塊的資訊
     166 === -locations ===
     167 * 印出每個區塊的位置
     168 === -racks ===
     169 * 印出data-node的網絡拓撲結構
     170
     171 如：
     172
     173$ bin/hadoop fsck /user/waue/input/1.txt -files -blocks -locations
     174
     175/user/waue/input/1.txt 115045564 bytes, 2 block(s):  Under replicated blk_-90085106852013388_1001. Target Replicas is 3 but found 2 replica(s).
     176
     177 Under replicated blk_-4027196261436469955_1001. Target Replicas is 3 but found 2 replica(s).
     178
     1790. blk_-90085106852013388_1001 len=67108864 repl=2 [140.110.138.191:50010, 140.110.141.129:50010]
     180
     1811. blk_-4027196261436469955_1001 len=47936700 repl=2 [140.110.138.191:50010, 140.110.141.129:50010]
     182
     183
     184
     185Status: HEALTHY
     186
     187 Total size:    115045564 B
     188
     189 Total dirs:    0
     190
     191 Total files:   1
     192