wiki:Hadoop_Lab4
實做四: Hadoop 程式編譯

範例練習 0 使用範例

$ cd /opt/hadoop
  • 下載 nchc-example.jar
    $ wget http://trac.nchc.org.tw/cloud/raw-attachment/wiki/Hadoop_Lab4/nchc-example.jar
    
  • 執行客製化的hadoop 程式
$ bin/hadoop jar nchc-example.jar
  • 輸出結果
    ******************************************
    歡迎使用 NCHC.Hadoop 的運算功能
    指令:
      Hadoop jar nchc-example-*.jar <功能>
    功能:
      hello:     印出key-value的內容
      wordcount: 計算輸入資料夾內分別在每個檔案的字數統計
      mwc:       統合計算所有輸入檔的字數統計
      grep:      算出包指定字串的個數
      nchcgrep:  整合來源檔內的每個字與其所有出現的所在列
    ******************************************
    

以 Hello 為例

code

  • 使用hello
    $ bin/hadoop jar nchc-example.jar hello
    
  • 使用提示:
    hello <inDir> <outDir> <m> <r>
    
  • 運作:
$ bin/hadoop dfs -put conf-local conf_input
$ bin/hadoop jar nchc-example.jar hello conf_input lab4_out6 1 1
  • 看結果

練習 1 : Word Count 編譯+執行

code

  • 上傳內容到hdfs內
$ cd /opt/hadoop
$ mkdir lab4_input
$ echo "I like NCHC Cloud Course." > lab4_input/input1
$ echo "I like nchc Cloud Course, and we enjoy this course." > lab4_input/input2
$ bin/hadoop fs -put lab4_input lab4_input
$ bin/hadoop fs -ls lab4_input
  • 下載 WordCount.java 並存到/opt/hadoop;
    $ wget http://trac.nchc.org.tw/cloud/raw-attachment/wiki/Hadoop_Lab4/WordCount.java
    
  • 編譯
$ mkdir MyJava
$ javac -classpath hadoop-*-core.jar -d MyJava WordCount.java
$ jar -cvf wordcount.jar -C MyJava .
  • 運算
    $ bin/hadoop jar wordcount.jar WordCount lab4_input/ lab4_out1/
    $ bin/hadoop fs -cat lab4_out1/part-r-00000
    
  • lab4_out1 執行結果
    Cloud 2
    Course, 1
    Course. 1
    I 2
    NCHC  1
    and 1
    course. 1
    enjoy 1
    like  2
    nchc  1
    this  1
    we  1
    

練習 2 : Word Count 進階版

code

$ echo "\." >pattern.txt && echo "\," >>pattern.txt
$ bin/hadoop fs -put pattern.txt ./
$ mkdir MyJava2
  • 下載 WordCount2.java 並存到/opt/hadoop;
    $ wget http://trac.nchc.org.tw/cloud/raw-attachment/wiki/Hadoop_Lab4/WordCount2.java
    
$ javac -classpath hadoop-*-core.jar -d MyJava2 WordCount2.java
$ jar -cvf wordcount2.jar -C MyJava2 .
$ bin/hadoop jar wordcount2.jar WordCount2 lab4_input lab4_out2 -skip pattern.txt
$ bin/hadoop fs -cat lab4_out2/part-00000
  • lab4_out2 執行結果
    Cloud 2
    Course  2
    I 2
    NCHC  1
    and 1
    course  1
    enjoy 1
    like  2
    nchc  1
    this  1
    we  1
    
$ bin/hadoop jar wordcount2.jar WordCount2 -Dwordcount.case.sensitive=false lab4_input lab4_out3 -skip pattern.txt
$ bin/hadoop fs -cat lab4_out3/part-00000
  • lab4_out3 執行結果
    and 1
    cloud 2
    course  3
    enjoy 1
    i 2
    like  2
    nchc  2
    this  1
    we  1
    
Last modified 8 years ago Last modified on Dec 19, 2011, 11:55:03 PM

Attachments (3)

Download all attachments as: .zip