Changes between Initial Version and Version 1 of NTUT160220/Lab5


Ignore:
Timestamp:
Feb 19, 2016, 9:57:37 PM (10 years ago)
Author:
jazz
Comment:

--

Legend:

Unmodified
Added
Removed
Modified

  • NTUT160220/Lab5

    v1 v1  
     1[[PageOutline]]
     2
     3◢ <[wiki:NTUT160220/Lab4 實作四]> | <[wiki:NTUT160220 回課程大綱]> ▲ | <[wiki:NTUT160220/Lab6 實作六]> ◣
     4
     5= 實作五 Lab 5 =
     6
     7{{{
     8#!html
     9<div style="text-align: center;"><big style="font-weight: bold;"><big>在單機模式執行 MapReduce 基本運算<br/>Running MapReduce in local mode by Examples</big></big></div>
     10}}}
     11
     12{{{
     13#!text
     14以下練習，請在本機的 Hadoop4Win 環境操作。
     15}}}
     16
     17== 範例一『字數統計(WordCount)』 ==
     18
     19 * STEP 1 : 練習 MapReduce 丟 Job 指令: 『__'''hadoop jar <local jar file> <class name> <parameters>'''__』
     20{{{
     21Jazz@human ~
     22$ cd /opt/hadoop/
     23
     24Jazz@human /opt/hadoop
     25$ hadoop jar hadoop-*-examples.jar wordcount input output
     2611/10/21 14:08:58 INFO input.FileInputFormat: Total input paths to process : 12
     2711/10/21 14:09:00 INFO mapred.JobClient: Running job: job_201110211130_0001
     2811/10/21 14:09:01 INFO mapred.JobClient:  map 0% reduce 0%
     2911/10/21 14:09:31 INFO mapred.JobClient:  map 16% reduce 0%
     3011/10/21 14:10:29 INFO mapred.JobClient:  map 100% reduce 27%
     3111/10/21 14:10:33 INFO mapred.JobClient:  map 100% reduce 100%
     3211/10/21 14:10:35 INFO mapred.JobClient: Job complete: job_201110211130_0001
     3311/10/21 14:10:35 INFO mapred.JobClient: Counters: 17
     3411/10/21 14:10:35 INFO mapred.JobClient:   Job Counters
     3511/10/21 14:10:35 INFO mapred.JobClient:     Launched reduce tasks=1
     3611/10/21 14:10:35 INFO mapred.JobClient:     Launched map tasks=12
     3711/10/21 14:10:35 INFO mapred.JobClient:     Data-local map tasks=12
     3811/10/21 14:10:35 INFO mapred.JobClient:   FileSystemCounters
     3911/10/21 14:10:35 INFO mapred.JobClient:     FILE_BYTES_READ=16578
     4011/10/21 14:10:35 INFO mapred.JobClient:     HDFS_BYTES_READ=18312
     4111/10/21 14:10:35 INFO mapred.JobClient:     FILE_BYTES_WRITTEN=32636
     4211/10/21 14:10:35 INFO mapred.JobClient:     HDFS_BYTES_WRITTEN=10922
     4311/10/21 14:10:35 INFO mapred.JobClient:   Map-Reduce Framework
     4411/10/21 14:10:35 INFO mapred.JobClient:     Reduce input groups=592
     4511/10/21 14:10:35 INFO mapred.JobClient:     Combine output records=750
     4611/10/21 14:10:35 INFO mapred.JobClient:     Map input records=553
     4711/10/21 14:10:35 INFO mapred.JobClient:     Reduce shuffle bytes=15674
     4811/10/21 14:10:35 INFO mapred.JobClient:     Reduce output records=592
     4911/10/21 14:10:35 INFO mapred.JobClient:     Spilled Records=1500
     5011/10/21 14:10:35 INFO mapred.JobClient:     Map output bytes=24438
     5111/10/21 14:10:35 INFO mapred.JobClient:     Combine input records=1755
     5211/10/21 14:10:35 INFO mapred.JobClient:     Map output records=1755
     5311/10/21 14:10:35 INFO mapred.JobClient:     Reduce input records=750
     54}}}
     55   * [[BR]][[Image(Hadoop4Win:hadoop4win_14.jpg,width=600)]]
     56
     57 * STEP 2 : 練習從 http://localhost:50030 查看目前 MapReduce Job 的運作情形
     58   * [[BR]][[Image(Hadoop4Win:hadoop4win_15.jpg,width=600)]]
     59
     60 * STEP 3 : 使用 HDFS 指令: 『__'''hadoop fs -get <HDFS file/dir> <local file/dir>'''__』，並了解輸出檔案檔名均為 part-r-*****，且執行參數會紀錄於 <HOSTNAME>_<TIME>_job_<JOBID>_0001_conf.xml，不妨可以觀察 xml 內容與 hadoop config 檔的參數關聯。
     61{{{
     62Jazz@human /opt/hadoop
     63$ hadoop fs -get output my_output
     64
     65Jazz@human /opt/hadoop
     66$ ls -alR my_output
     67my_output:
     68total 12
     69drwxr-xr-x+  3 Jazz None     0 Oct 21 14:12 .
     70drwxr-xr-x+ 15 Jazz None     0 Oct 21 14:12 ..
     71drwxr-xr-x+  3 Jazz None     0 Oct 21 14:12 _logs
     72-rwxr-xr-x   1 Jazz None 10922 Oct 21 14:12 part-r-00000
     73
     74my_output/_logs:
     75total 0
     76drwxr-xr-x+ 3 Jazz None 0 Oct 21 14:12 .
     77drwxr-xr-x+ 3 Jazz None 0 Oct 21 14:12 ..
     78drwxr-xr-x+ 2 Jazz None 0 Oct 21 14:12 history
     79
     80my_output/_logs/history:
     81total 48
     82drwxr-xr-x+ 2 Jazz None     0 Oct 21 14:12 .
     83drwxr-xr-x+ 3 Jazz None     0 Oct 21 14:12 ..
     84-rwxr-xr-x  1 Jazz None 26004 Oct 21 14:12 localhost_1319167815125_job_201110211130_0001_Jazz_word+count
     85-rwxr-xr-x  1 Jazz None 16984 Oct 21 14:12 localhost_1319167815125_job_201110211130_0001_conf.xml
     86}}}
     87   * [[BR]][[Image(Hadoop4Win:hadoop4win_22.jpg,width=600)]]
     88
     89 * 比較熟悉 Windows 的,也可以用 cgystart my_output 用檔案總管開啟下載的內容
     90{{{
     91Jazz@human /opt/hadoop
     92$ cgystart my_output
     93}}}
     94
     95 * 也可以透過 NameNode 網頁介面觀看結果
     96   * http://localhost:50075/browseDirectory.jsp?dir=/user&namenodeInfoPort=50070
     97
     98== 範例二『用標準表示法過濾內容 grep』 ==
     99
     100 * grep 這個命令是擷取文件裡面特定的字元，在 Hadoop example 中此指令可以擷取文件中有此指定文字的字串，並作計數統計[[BR]]grep is a command to extract specific characters in documents. In hadoop examples, you can use this command to extract strings match the regular expression and count for matched strings.
     101{{{
     102Jazz@human /opt/hadoop
     103$ hadoop jar hadoop-*-examples.jar  grep input lab5_out1 'dfs[a-z.]+'
     104}}}
     105 * 運作的畫面如下：[[BR]]You should see procedure like this:
     106{{{
     107Jazz@human /opt/hadoop
     108$ hadoop jar hadoop-*-examples.jar  grep input lab5_out1 'dfs[a-z.]+'
     10911/10/21 14:17:39 INFO mapred.FileInputFormat: Total input paths to process : 12
     110
     11111/10/21 14:17:39 INFO mapred.JobClient: Running job: job_201110211130_0002
     11211/10/21 14:17:40 INFO mapred.JobClient:  map 0% reduce 0%
     11311/10/21 14:17:54 INFO mapred.JobClient:  map 8% reduce 0%
     11411/10/21 14:17:57 INFO mapred.JobClient:  map 16% reduce 0%
     11511/10/21 14:18:03 INFO mapred.JobClient:  map 33% reduce 0%
     11611/10/21 14:18:13 INFO mapred.JobClient:  map 41% reduce 0%
     11711/10/21 14:18:16 INFO mapred.JobClient:  map 50% reduce 11%
     11811/10/21 14:18:19 INFO mapred.JobClient:  map 58% reduce 11%
     11911/10/21 14:18:23 INFO mapred.JobClient:  map 66% reduce 11%
     12011/10/21 14:18:30 INFO mapred.JobClient:  map 83% reduce 16%
     12111/10/21 14:18:33 INFO mapred.JobClient:  map 83% reduce 22%
     12211/10/21 14:18:36 INFO mapred.JobClient:  map 91% reduce 22%
     12311/10/21 14:18:39 INFO mapred.JobClient:  map 100% reduce 22%
     12411/10/21 14:18:42 INFO mapred.JobClient:  map 100% reduce 27%
     12511/10/21 14:18:48 INFO mapred.JobClient:  map 100% reduce 30%
     12611/10/21 14:18:54 INFO mapred.JobClient:  map 100% reduce 100%
     12711/10/21 14:18:56 INFO mapred.JobClient: Job complete: job_201110211130_0002
     12811/10/21 14:18:56 INFO mapred.JobClient: Counters: 18
     12911/10/21 14:18:56 INFO mapred.JobClient:   Job Counters
     13011/10/21 14:18:56 INFO mapred.JobClient:     Launched reduce tasks=1
     13111/10/21 14:18:56 INFO mapred.JobClient:     Launched map tasks=12
     13211/10/21 14:18:56 INFO mapred.JobClient:     Data-local map tasks=12
     13311/10/21 14:18:56 INFO mapred.JobClient:   FileSystemCounters
     13411/10/21 14:18:56 INFO mapred.JobClient:     FILE_BYTES_READ=888
     13511/10/21 14:18:56 INFO mapred.JobClient:     HDFS_BYTES_READ=18312
     13611/10/21 14:18:56 INFO mapred.JobClient:     FILE_BYTES_WRITTEN=1496
     13711/10/21 14:18:56 INFO mapred.JobClient:     HDFS_BYTES_WRITTEN=280
     13811/10/21 14:18:56 INFO mapred.JobClient:   Map-Reduce Framework
     13911/10/21 14:18:56 INFO mapred.JobClient:     Reduce input groups=7
     14011/10/21 14:18:56 INFO mapred.JobClient:     Combine output records=7
     14111/10/21 14:18:56 INFO mapred.JobClient:     Map input records=553
     14211/10/21 14:18:56 INFO mapred.JobClient:     Reduce shuffle bytes=224
     14311/10/21 14:18:56 INFO mapred.JobClient:     Reduce output records=7
     14411/10/21 14:18:56 INFO mapred.JobClient:     Spilled Records=14
     14511/10/21 14:18:56 INFO mapred.JobClient:     Map output bytes=193
     14611/10/21 14:18:56 INFO mapred.JobClient:     Map input bytes=18312
     14711/10/21 14:18:56 INFO mapred.JobClient:     Combine input records=10
     14811/10/21 14:18:56 INFO mapred.JobClient:     Map output records=10
     14911/10/21 14:18:56 INFO mapred.JobClient:     Reduce input records=7
     15011/10/21 14:18:56 WARN mapred.JobClient: Use GenericOptionsParser for parsing th
     151e arguments. Applications should implement Tool for the same.
     15211/10/21 14:18:57 INFO mapred.FileInputFormat: Total input paths to process : 1
     15311/10/21 14:18:57 INFO mapred.JobClient: Running job: job_201110211130_0003
     154( ... skip ... )
     155}}}
     156 * 接著查看結果[[BR]]Let's check the computed result of '''grep''' from HDFS :
     157 * 這個例子是要從 input 目錄中的所有檔案中找出符合 dfs 後面接著 a-z 字母一個以上的字串
     158{{{
     159Jazz@human /opt/hadoop
     160$ hadoop fs -ls lab5_out1
     161Found 2 items
     162drwxr-xr-x   - Jazz supergroup          0 2011-10-21 14:18 /user/Jazz/lab5_out1/_logs
     163-rw-r--r--   1 Jazz supergroup         96 2011-10-21 14:19 /user/Jazz/lab5_out1/part-00000
     164
     165Jazz@human /opt/hadoop
     166$ hadoop fs -cat lab5_out1/part-00000
     1673       dfs.class
     1682       dfs.period
     1691       dfs.file
     1701       dfs.replication
     1711       dfs.servers
     1721       dfsadmin
     1731       dfsmetrics.log
     174}}}