| 1 | {{{ |
| 2 | #!html |
| 3 | <div style="text-align: center;"><big |
| 4 | style="font-weight: bold;"><big><big>實做三、執行 MapReduce 基本運算</big></big></big></div> |
| 5 | }}} |
| 6 | [[PageOutline]] |
| 7 | |
| 8 | == 1 Hadoop運算命令 grep == |
| 9 | |
| 10 | * grep 這個命令是擷取文件裡面特定的字元,在Hadoop example中此指令可以擷取文件中有此指定文字的字串,並作計數統計 |
| 11 | |
| 12 | {{{ |
| 13 | /opt/hadoop$ bin/hadoop jar hadoop-*-examples.jar grep input grep_output 'dfs[a-z.]+' |
| 14 | |
| 15 | }}} |
| 16 | |
| 17 | 運作的畫面如下: |
| 18 | |
| 19 | {{{ |
| 20 | |
| 21 | 09/03/24 12:33:45 INFO mapred.FileInputFormat: Total input paths to process : 9 |
| 22 | 09/03/24 12:33:45 INFO mapred.FileInputFormat: Total input paths to process : 9 |
| 23 | 09/03/24 12:33:45 INFO mapred.JobClient: Running job: job_200903232025_0003 |
| 24 | 09/03/24 12:33:46 INFO mapred.JobClient: map 0% reduce 0% |
| 25 | 09/03/24 12:33:47 INFO mapred.JobClient: map 10% reduce 0% |
| 26 | 09/03/24 12:33:49 INFO mapred.JobClient: map 20% reduce 0% |
| 27 | 09/03/24 12:33:51 INFO mapred.JobClient: map 30% reduce 0% |
| 28 | 09/03/24 12:33:52 INFO mapred.JobClient: map 40% reduce 0% |
| 29 | 09/03/24 12:33:54 INFO mapred.JobClient: map 50% reduce 0% |
| 30 | 09/03/24 12:33:55 INFO mapred.JobClient: map 60% reduce 0% |
| 31 | 09/03/24 12:33:57 INFO mapred.JobClient: map 70% reduce 0% |
| 32 | 09/03/24 12:33:59 INFO mapred.JobClient: map 80% reduce 0% |
| 33 | 09/03/24 12:34:00 INFO mapred.JobClient: map 90% reduce 0% |
| 34 | 09/03/24 12:34:02 INFO mapred.JobClient: map 100% reduce 0% |
| 35 | 09/03/24 12:34:10 INFO mapred.JobClient: map 100% reduce 10% |
| 36 | 09/03/24 12:34:12 INFO mapred.JobClient: map 100% reduce 13% |
| 37 | 09/03/24 12:34:15 INFO mapred.JobClient: map 100% reduce 20% |
| 38 | 09/03/24 12:34:20 INFO mapred.JobClient: map 100% reduce 23% |
| 39 | 09/03/24 12:34:22 INFO mapred.JobClient: Job complete: job_200903232025_0003 |
| 40 | 09/03/24 12:34:22 INFO mapred.JobClient: Counters: 16 |
| 41 | 09/03/24 12:34:22 INFO mapred.JobClient: File Systems |
| 42 | 09/03/24 12:34:22 INFO mapred.JobClient: HDFS bytes read=48245 |
| 43 | 09/03/24 12:34:22 INFO mapred.JobClient: HDFS bytes written=1907 |
| 44 | 09/03/24 12:34:22 INFO mapred.JobClient: Local bytes read=1549 |
| 45 | 09/03/24 12:34:22 INFO mapred.JobClient: Local bytes written=3584 |
| 46 | 09/03/24 12:34:22 INFO mapred.JobClient: Job Counters |
| 47 | ...... |
| 48 | }}} |
| 49 | |
| 50 | |
| 51 | * 接著查看結果 |
| 52 | |
| 53 | {{{ |
| 54 | /opt/hadoop$ bin/hadoop fs -ls grep_output |
| 55 | /opt/hadoop$ bin/hadoop fs -cat grep_output/part-00000 |
| 56 | }}} |
| 57 | |
| 58 | 結果如下 |
| 59 | |
| 60 | {{{ |
| 61 | 3 dfs.class |
| 62 | 3 dfs. |
| 63 | 2 dfs.period |
| 64 | 1 dfs.http.address |
| 65 | 1 dfs.balance.bandwidth |
| 66 | 1 dfs.block.size |
| 67 | 1 dfs.blockreport.initial |
| 68 | 1 dfs.blockreport.interval |
| 69 | 1 dfs.client.block.write.retries |
| 70 | 1 dfs.client.buffer.dir |
| 71 | 1 dfs.data.dir |
| 72 | 1 dfs.datanode.address |
| 73 | 1 dfs.datanode.dns.interface |
| 74 | 1 dfs.datanode.dns.nameserver |
| 75 | 1 dfs.datanode.du.pct |
| 76 | 1 dfs.datanode.du.reserved |
| 77 | 1 dfs.datanode.handler.count |
| 78 | 1 dfs.datanode.http.address |
| 79 | 1 dfs.datanode.https.address |
| 80 | 1 dfs.datanode.ipc.address |
| 81 | 1 dfs.default.chunk.view.size |
| 82 | 1 dfs.df.interval |
| 83 | 1 dfs.file |
| 84 | 1 dfs.heartbeat.interval |
| 85 | 1 dfs.hosts |
| 86 | 1 dfs.hosts.exclude |
| 87 | 1 dfs.https.address |
| 88 | 1 dfs.impl |
| 89 | 1 dfs.max.objects |
| 90 | 1 dfs.name.dir |
| 91 | 1 dfs.namenode.decommission.interval |
| 92 | 1 dfs.namenode.decommission.interval. |
| 93 | 1 dfs.namenode.decommission.nodes.per.interval |
| 94 | 1 dfs.namenode.handler.count |
| 95 | 1 dfs.namenode.logging.level |
| 96 | 1 dfs.permissions |
| 97 | 1 dfs.permissions.supergroup |
| 98 | 1 dfs.replication |
| 99 | 1 dfs.replication.consider |
| 100 | 1 dfs.replication.interval |
| 101 | 1 dfs.replication.max |
| 102 | 1 dfs.replication.min |
| 103 | 1 dfs.replication.min. |
| 104 | 1 dfs.safemode.extension |
| 105 | 1 dfs.safemode.threshold.pct |
| 106 | 1 dfs.secondary.http.address |
| 107 | 1 dfs.servers |
| 108 | 1 dfs.web.ugi |
| 109 | 1 dfsmetrics.log |
| 110 | |
| 111 | }}} |
| 112 | |
| 113 | == 2 Hadoop運算命令 WordCount == |
| 114 | |
| 115 | * 如名稱,WordCount會對所有的字作字數統計,並且從a-z作排列 |
| 116 | |
| 117 | {{{ |
| 118 | /opt/hadoop$ bin/hadoop jar hadoop-*-examples.jar wordcount input wc_output |
| 119 | }}} |
| 120 | |
| 121 | 檢查輸出結果的方法同2.1的方法 |
| 122 | |
| 123 | === 2.1 更多運算命令 === |
| 124 | |
| 125 | 可執行的指令一覽表: |
| 126 | |
| 127 | || aggregatewordcount || An Aggregate based map/reduce program that counts the words in the input files. || |
| 128 | || aggregatewordhist || An Aggregate based map/reduce program that computes the histogram of the words in the input files. || |
| 129 | || grep || A map/reduce program that counts the matches of a regex in the input. || |
| 130 | || join || A job that effects a join over sorted, equally partitioned datasets || |
| 131 | || multifilewc || A job that counts words from several files. || |
| 132 | || pentomino || A map/reduce tile laying program to find solutions to pentomino problems. || |
| 133 | || pi || A map/reduce program that estimates Pi using monte-carlo method. || |
| 134 | || randomtextwriter || A map/reduce program that writes 10GB of random textual data per node. || |
| 135 | || randomwriter || A map/reduce program that writes 10GB of random data per node. || |
| 136 | || sleep || A job that sleeps at each map and reduce task. || |
| 137 | || sort || A map/reduce program that sorts the data written by the random writer. || |
| 138 | || sudoku || A sudoku solver. || |
| 139 | || wordcount || A map/reduce program that counts the words in the input files. || |
| 140 | |
| 141 | 請參考 [http://hadoop.apache.org/core/docs/current/api/org/apache/hadoop/examples/package-summary.html org.apache.hadoop.examples] |
| 142 | |
| 143 | |
| 144 | == 3. 使用網頁Gui瀏覽資訊 == |
| 145 | |
| 146 | * [http://localhost:50030 Map/Reduce Administration] |
| 147 | * [http://localhost:50070 NameNode ] |