Changes between Version 15 and Version 16 of jazz/Hadoop_Lab2

Timestamp:

Mar 24, 2009, 2:10:07 PM (15 years ago)

Author:

waue

Comment:

--

jazz/Hadoop_Lab2

@@ -9 +9 @@
  == Content 1. 基本操作 ==
  === 1.1 瀏覽你HDFS目錄  === 
-
+{{{
+/opt/hadoop$ bin/hadoop fs -ls
+}}}
  === 1.2 上傳資料到HDFS目錄 === 
+ * 上傳
+{{{
+/opt/hadoop$ bin/hadoop fs -put conf input
+}}} 
+ * 檢查
+{{{
+/opt/hadoop$ bin/hadoop fs -ls
+/opt/hadoop$ bin/hadoop fs -ls input
+}}}
  
  === 1.3 下載HDFS的資料到本地目錄 === 
+ * 下載
+{{{
+/opt/hadoop$ bin/hadoop fs -get input fromHDFS
+}}} 
+ * 檢查
+{{{
+/opt/hadoop$ ls -al | grep fromHDFS
+/opt/hadoop$ ls -al fromHDFS
+}}}  
+
+ === 1.4 刪除檔案 ===
+{{{
+/opt/hadoop$ bin/hadoop fs -ls input
+/opt/hadoop$ bin/hadoop fs -rm input/masters
+}}} 
+ === 1.5 直接看檔案 ===
+{{{
+/opt/hadoop$ bin/hadoop fs -ls input
+/opt/hadoop$ bin/hadoop fs -cat input/slaves
+}}} 
+
+ === 1.6 更多指令操作 ===
+{{{
+hadooper@vPro:/opt/hadoop$ bin/hadoop fs 
+
+Usage: java FsShell
+
+           [-ls <path>]
+
+           [-lsr <path>]
+
+           [-du <path>]
+
+           [-dus <path>]
+
+           [-count[-q] <path>]
+
+           [-mv <src> <dst>]
+
+           [-cp <src> <dst>]
+
+           [-rm <path>]
+
+           [-rmr <path>]
+
+           [-expunge]
+
+           [-put <localsrc> ... <dst>]
+
+           [-copyFromLocal <localsrc> ... <dst>]
+
+           [-moveFromLocal <localsrc> ... <dst>]
+
+           [-get [-ignoreCrc] [-crc] <src> <localdst>]
+
+           [-getmerge <src> <localdst> [addnl]]
+
+           [-cat <src>]
+
+           [-text <src>]
+
+           [-copyToLocal [-ignoreCrc] [-crc] <src> <localdst>]
+
+           [-moveToLocal [-crc] <src> <localdst>]
+
+           [-mkdir <path>]
+
+           [-setrep [-R] [-w] <rep> <path/file>]
+
+           [-touchz <path>]
+
+           [-test -[ezd] <path>]
+
+           [-stat [format] <path>]
+
+           [-tail [-f] <file>]
+
+           [-chmod [-R] <MODE[,MODE]... | OCTALMODE> PATH...]
+
+           [-chown [-R] [OWNER][:[GROUP]] PATH...]
+
+           [-chgrp [-R] GROUP PATH...]
+
+           [-help [cmd]]
+
+
+
+Generic options supported are
+
+-conf <configuration file>     specify an application configuration file
+
+-D <property=value>            use value for given property
+
+-fs <local|namenode:port>      specify a namenode
+
+-jt <local|jobtracker:port>    specify a job tracker
+
+-files <comma separated list of files>    specify comma separated files to be copied to the map reduce cluster
+
+-libjars <comma separated list of jars>    specify comma separated jar files to include in the classpath.
+
+-archives <comma separated list of archives>    specify comma separated archives to be unarchived on the compute machines.
+
+
+
+The general command line syntax is
+
+bin/hadoop command [genericOptions] [commandOptions]
+
+
+}}}  
   
- === 1.4 更多指令操作 ===
+ === 1.7 自我練習 ===
+ 
+ * 刪除在 hdfs 內的一整個的資料夾 input
   
  == Content 2. Hadoop 運算命令 == 
@@ -20 +144 @@
  === 2.1 Hadoop運算命令 grep ===  
  
+ * grep 這個命令是擷取文件裡面特定的字元，在Hadoop example中此指令可以擷取文件中有此指定文字的字串，並作計數統計
+ 
+ {{{
+ /opt/hadoop$ bin/hadoop jar hadoop-*-examples.jar grep input grep_output 'dfs[a-z.]+' 
+ }}}
+ 
+ 運作的畫面如下：
+ 
+ {{{
+09/03/24 12:33:45 INFO mapred.FileInputFormat: Total input paths to process : 9
+
+09/03/24 12:33:45 INFO mapred.FileInputFormat: Total input paths to process : 9
+
+09/03/24 12:33:45 INFO mapred.JobClient: Running job: job_200903232025_0003
+
+09/03/24 12:33:46 INFO mapred.JobClient:  map 0% reduce 0%
+
+09/03/24 12:33:47 INFO mapred.JobClient:  map 10% reduce 0%
+
+09/03/24 12:33:49 INFO mapred.JobClient:  map 20% reduce 0%
+
+09/03/24 12:33:51 INFO mapred.JobClient:  map 30% reduce 0%
+
+09/03/24 12:33:52 INFO mapred.JobClient:  map 40% reduce 0%
+
+09/03/24 12:33:54 INFO mapred.JobClient:  map 50% reduce 0%
+
+09/03/24 12:33:55 INFO mapred.JobClient:  map 60% reduce 0%
+
+09/03/24 12:33:57 INFO mapred.JobClient:  map 70% reduce 0%
+
+09/03/24 12:33:59 INFO mapred.JobClient:  map 80% reduce 0%
+
+09/03/24 12:34:00 INFO mapred.JobClient:  map 90% reduce 0%
+
+09/03/24 12:34:02 INFO mapred.JobClient:  map 100% reduce 0%
+
+09/03/24 12:34:10 INFO mapred.JobClient:  map 100% reduce 10%
+
+09/03/24 12:34:12 INFO mapred.JobClient:  map 100% reduce 13%
+
+09/03/24 12:34:15 INFO mapred.JobClient:  map 100% reduce 20%
+
+09/03/24 12:34:20 INFO mapred.JobClient:  map 100% reduce 23%
+
+09/03/24 12:34:22 INFO mapred.JobClient: Job complete: job_200903232025_0003
+
+09/03/24 12:34:22 INFO mapred.JobClient: Counters: 16
+
+09/03/24 12:34:22 INFO mapred.JobClient:   File Systems
+
+09/03/24 12:34:22 INFO mapred.JobClient:     HDFS bytes read=48245
+
+09/03/24 12:34:22 INFO mapred.JobClient:     HDFS bytes written=1907
+
+09/03/24 12:34:22 INFO mapred.JobClient:     Local bytes read=1549
+
+09/03/24 12:34:22 INFO mapred.JobClient:     Local bytes written=3584
+
+09/03/24 12:34:22 INFO mapred.JobClient:   Job Counters 
+
+......
+ }}}
+ 
+ * 接著查看結果
+ {{{
+  /opt/hadoop$ bin/hadoop fs -ls grep_output
+  /opt/hadoop$ bin/hadoop fs -cat grep_output/part-00000 
+ }}}
+ 結果如下
+ {{{
+
+3       dfs.class
+
+3       dfs.
+
+2       dfs.period
+
+1       dfs.http.address
+
+1       dfs.balance.bandwidth
+
+1       dfs.block.size
+
+1       dfs.blockreport.initial
+
+1       dfs.blockreport.interval
+
+1       dfs.client.block.write.retries
+
+1       dfs.client.buffer.dir
+
+1       dfs.data.dir
+
+1       dfs.datanode.address
+
+1       dfs.datanode.dns.interface
+
+1       dfs.datanode.dns.nameserver
+
+1       dfs.datanode.du.pct
+
+1       dfs.datanode.du.reserved
+
+1       dfs.datanode.handler.count
+
+1       dfs.datanode.http.address
+
+1       dfs.datanode.https.address
+
+1       dfs.datanode.ipc.address
+
+1       dfs.default.chunk.view.size
+
+1       dfs.df.interval
+
+1       dfs.file
+
+1       dfs.heartbeat.interval
+
+1       dfs.hosts
+
+1       dfs.hosts.exclude
+
+1       dfs.https.address
+
+1       dfs.impl
+
+1       dfs.max.objects
+
+1       dfs.name.dir
+
+1       dfs.namenode.decommission.interval
+
+1       dfs.namenode.decommission.interval.
+
+1       dfs.namenode.decommission.nodes.per.interval
+
+1       dfs.namenode.handler.count
+
+1       dfs.namenode.logging.level
+
+1       dfs.permissions
+
+1       dfs.permissions.supergroup
+
+1       dfs.replication
+
+1       dfs.replication.consider
+
+1       dfs.replication.interval
+
+1       dfs.replication.max
+
+1       dfs.replication.min
+
+1       dfs.replication.min.
+
+1       dfs.safemode.extension
+
+1       dfs.safemode.threshold.pct
+
+1       dfs.secondary.http.address
+
+1       dfs.servers
+
+1       dfs.web.ugi
+
+1       dfsmetrics.log
+
+
+ }}}
  === 2.2 Hadoop運算命令 WordCount ===
  
+ * 如名稱，WordCount會對所有的字作字數統計，並且從a-z作排列
+ 
+ {{{
+ /opt/hadoop$ bin/hadoop jar hadoop-*-examples.jar wordcount input wc_output
+ }}}
+ 
+ 檢查輸出結果的方法同2.1的方法
+
  === 2.3 更多運算命令 ===
  
@@ -40 +344 @@
  || wordcount || A map/reduce program that counts the words in the input files. || 
 
- 
  請參考 [http://hadoop.apache.org/core/docs/current/api/org/apache/hadoop/examples/package-summary.html org.apache.hadoop.examples]
- 
-
- 
- == Content 3. 使用網頁Gui瀏覽訊息 ==
- 
- 
- == 練習 == 
- 
- 
+
+{{{
+#!html
+<html lang="zh-tw"><head>
+
+<meta content="text/html; charset=ISO-8859-1" http-equiv="content-type"><title>a.html</title>
+
+</head><body>
+<br>
+
+<p>
+</p><table summary="" border="1" cellpadding="3" cellspacing="0" width="100%">
+<tbody><tr class="TableHeadingColor" bgcolor="#ccccff">
+<th colspan="2" align="left"><font size="+2">
+<b>Class Summary</b></font></th>
+</tr>
+<tr class="TableRowColor" bgcolor="white">
+
+<td width="15%"><b><a href="http://hadoop.apache.org/core/docs/current/api/org/apache/hadoop/examples/AggregateWordCount.html" title="class in org.apache.hadoop.examples">AggregateWordCount</a></b></td>
+<td>This is an example Aggregated Hadoop Map/Reduce application. It
+reads the text input files, breaks each line into words and counts
+them. The output is a locally sorted list of words and the count of how
+often they occurred. To run: bin/hadoop jar hadoop-*-examples.jar
+aggregatewordcount in-dir out-dir numOfReducers textinputformat </td>
+</tr>
+<tr class="TableRowColor" bgcolor="white">
+<td width="15%"><b><a href="http://hadoop.apache.org/core/docs/current/api/org/apache/hadoop/examples/AggregateWordHistogram.html" title="class in org.apache.hadoop.examples">AggregateWordHistogram</a></b></td>
+<td>This is an example Aggregated Hadoop Map/Reduce application.
+Computes the histogram of the words in the input texts. To run:
+bin/hadoop jar hadoop-*-examples.jar aggregatewordhist in-dir out-dir
+numOfReducers textinputformat </td>
+</tr>
+<tr class="TableRowColor" bgcolor="white">
+<td width="15%"><b><a href="http://hadoop.apache.org/core/docs/current/api/org/apache/hadoop/examples/ExampleDriver.html" title="class in org.apache.hadoop.examples">ExampleDriver</a></b></td>
+<td>A description of an example program based on its class and a human-readable description.</td>
+</tr>
+
+<tr class="TableRowColor" bgcolor="white">
+<td width="15%"><b><a href="http://hadoop.apache.org/core/docs/current/api/org/apache/hadoop/examples/Grep.html" title="class in org.apache.hadoop.examples">Grep</a></b></td>
+<td>&nbsp;</td>
+</tr>
+<tr class="TableRowColor" bgcolor="white">
+<td width="15%"><b><a href="http://hadoop.apache.org/core/docs/current/api/org/apache/hadoop/examples/Join.html" title="class in org.apache.hadoop.examples">Join</a></b></td>
+<td>This is the trivial map/reduce program that does absolutely nothing
+other than use the framework to fragment and sort the input values. To
+run: bin/hadoop jar build/hadoop-examples.jar join [-m maps] [-r
+reduces] [-inFormat input format class] [-outFormat output format
+class] [-outKey output key class] [-outValue output value class]
+[-joinOp <inner |outer|override="">] [in-dir]* in-dir out-dir</inner></td>
+</tr>
+<tr class="TableRowColor" bgcolor="white">
+<td width="15%"><b><a href="http://hadoop.apache.org/core/docs/current/api/org/apache/hadoop/examples/RandomTextWriter.html" title="class in org.apache.hadoop.examples">RandomTextWriter</a></b></td>
+<td>This program uses map/reduce to just run a distributed job where
+there is
+no interaction between the tasks and each task writes a large unsorted
+random sequence of words.To run: bin/hadoop jar
+hadoop-${version}-examples.jar randomtextwriter [-outFormat output
+format class] output</td>
+
+</tr>
+<tr class="TableRowColor" bgcolor="white">
+<td width="15%"><b><a href="http://hadoop.apache.org/core/docs/current/api/org/apache/hadoop/examples/RandomWriter.html" title="class in org.apache.hadoop.examples">RandomWriter</a></b></td>
+<td>This program uses map/reduce to just run a distributed job where
+there is
+no interaction between the tasks and each task write a large unsorted
+random binary sequence file of BytesWritable.To run: bin/hadoop jar
+hadoop-${version}-examples.jar randomwriter [-outFormat output format
+class] output</td>
+</tr>
+<tr class="TableRowColor" bgcolor="white">
+<td width="15%"><b><a href="http://hadoop.apache.org/core/docs/current/api/org/apache/hadoop/examples/Sort.html" title="class in org.apache.hadoop.examples">Sort&lt;K,V&gt;</a></b></td>
+<td>This is the trivial map/reduce program that does absolutely nothing
+other than use the framework to fragment and sort the input values.To
+run: bin/hadoop jar build/hadoop-examples.jar sort [-m maps] [-r
+reduces] [-inFormat input format class] [-outFormat output format
+class] [-outKey output key class] [-outValue output value class]
+[-totalOrder pcnt num samples max splits] in-dir out-dir</td>
+</tr>
+<tr class="TableRowColor" bgcolor="white">
+<td width="15%"><b><a href="http://hadoop.apache.org/core/docs/current/api/org/apache/hadoop/examples/WordCount.html" title="class in org.apache.hadoop.examples">WordCount</a></b></td>
+
+<td>This is an example Hadoop Map/Reduce application.</td>
+</tr>
+</tbody></table>
+</body></html>
+}}} 
+ 
+ === 2.4 練習 ===
+ 
+ 
+ == Content 3. 使用網頁Gui瀏覽資訊 ==
+ 
+ * [http://localhost:50030 Map/Reduce Administration]
+ * [http://localhost:50070 NameNode ]
+ 
+ === 3.1 練習 ===
+ 
+ * 用網頁秀出你在 wordcount練習的輸出結果
+