wiki:waue/Eclipse_Hadoop_JAR
close Warning: Can't synchronize with repository "(default)" (Unsupported version control system "svn": libffi.so.6: failed to map segment from shared object: Cannot allocate memory). Look in the Trac log for more information.

Version 5 (modified by waue, 17 years ago) (diff)

--

用Eclipse製成可在Hadoop上運行MapReduce的jar檔

ps : 需eclipse 3.3 以上 搭配 hadoop 0.17 以上版本。

  • 本篇的安裝環境是
名稱 目錄
Hadoop 安裝目錄 /opt/hadoop
來源資料夾 /opt/hadoop/input
輸出資料夾 /opt/hadoop/output
  1. 開啟MapReduce 專案
視窗操作 介面中設定 註解
File > new > Map/Reduce? Project>next Project name:sample
Configure Hadoop install directory: /opt/hadoop
=> Finish 		完成會增加sample專案並切換成MapReduce的視野
  1. 加入檔案WordCount.java檔
視窗操作 介面中設定 結果
右鍵點選sample專案 > new > file sample >src
File Name: WordCount.java
=> Finish 		完成後就多了一個WordCount.java檔
  1. 寫入WordCount.java的內容（code）
  1. 執行
視窗操作 介面中設定 結果
run > Run Configurations... Main tag :
Name: WordCount
Project: sample
Main class:: WordCount ;Arguments tag :
Program arguments: /opt/hadoop/log /opt/hadoop/test2 => Apply => Run 		console 介面會出現執行結果
  • Eclipse是用模擬的方式模擬Hadoop的環境，執行這段程式碼，所以並沒有送上HDFS給Hadoop的job tracker作Map Reduce。​http://localhost:50030 沒有工作運作的紀錄可以證明這點。
    • 既然是在本機端上運作，所以給的Program arguments參數 /opt/hadoop/input /opt/hadoop/output 是本機上的目錄。
    • 請確認 input 資料夾內有純文字資料，且output資料夾尚未存在（執行後系統會自行建立此資料夾並將結果放入）
  • 若Console 介面沒有錯誤訊息，則代表這段程式在主機端運作無誤 
    09/02/06 17:18:35 INFO jvm.JvmMetrics: Initializing JVM Metrics with processName=JobTracker, sessionId=
09/02/06 17:18:35 WARN mapred.JobClient: Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same.
09/02/06 17:18:35 WARN mapred.JobClient: No job jar file set.  User classes may not be found. See JobConf(Class) or JobConf#setJar(String).
09/02/06 17:18:35 INFO mapred.FileInputFormat: Total input paths to process : 1

... 略 ...

09/02/06 17:18:36 INFO mapred.JobClient:     Map output bytes=445846
09/02/06 17:18:36 INFO mapred.JobClient:     Map input bytes=320950
09/02/06 17:18:36 INFO mapred.JobClient:     Combine input records=37943
09/02/06 17:18:36 INFO mapred.JobClient:     Map output records=37943
09/02/06 17:18:36 INFO mapred.JobClient:     Reduce input records=9284

錯誤排除 ：

  • input 資料夾內有純文字資料
  • output 資料夾尚未存在（執行後系統會自行建立此資料夾並將結果放入）
  • 檢查"run configuration" 內的 "Java Application" > "WordCount" 的設定是否正確
  1. 打包成JAR
視窗操作 介面中設定 結果
File > Export > Java > Runnable JAR file Launch configuration : WordCount - sample
Export destionation : /opt/hadoop/WordCount.jar => Finish => ok 		/opt/hadoop/下可以找到檔案WordCount.jar
  • 最後一個ok在於包入Hadoop的必要library，所以匯出的WordCount.jar 檔大約有4.3MB
  1. 運行WordCount於HDFS之上

指令： 

$ cd /opt/hadoop
$ bin/hadoop jar WordCount.jar /user/waue/input /user/waue/out/
  • bin/hadoop jar 不可用 -jar，但若是單純用java執行jar， 則要用$ java -jar XXX.jar，不可只用jar
  • /user/waue/input /user/waue/out/ 為輸入和輸出的兩個參數，這兩個路徑是HDFS上得路徑，請確認hdfs內的/user/waue/input有純文字檔，且無/user/waue/out/這個資料夾。
  • 若已經成功執行過，想再執行第二次，請更換output的資料夾名稱，否則會因資料夾已存在而出現錯誤訊息。

執行畫面 

09/02/06 18:13:14 WARN mapred.JobClient: Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same.

09/02/06 18:13:14 INFO mapred.FileInputFormat: Total input paths to process : 1

09/02/06 18:13:14 INFO mapred.FileInputFormat: Total input paths to process : 1

09/02/06 18:13:15 INFO mapred.JobClient: Running job: job_200902051032_0009

09/02/06 18:13:16 INFO mapred.JobClient:  map 0% reduce 0%

09/02/06 18:13:20 INFO mapred.JobClient:  map 100% reduce 0%

09/02/06 18:13:23 INFO mapred.JobClient: Job complete: job_200902051032_0009

09/02/06 18:13:23 INFO mapred.JobClient: Counters: 16

09/02/06 18:13:23 INFO mapred.JobClient:   File Systems

09/02/06 18:13:23 INFO mapred.JobClient:     HDFS bytes read=320950

09/02/06 18:13:23 INFO mapred.JobClient:     HDFS bytes written=130568

09/02/06 18:13:23 INFO mapred.JobClient:     Local bytes read=168448

09/02/06 18:13:23 INFO mapred.JobClient:     Local bytes written=336932

09/02/06 18:13:23 INFO mapred.JobClient:   Job Counters 

09/02/06 18:13:23 INFO mapred.JobClient:     Launched reduce tasks=1

09/02/06 18:13:23 INFO mapred.JobClient:     Launched map tasks=1

09/02/06 18:13:23 INFO mapred.JobClient:     Data-local map tasks=1

09/02/06 18:13:23 INFO mapred.JobClient:   Map-Reduce Framework

09/02/06 18:13:23 INFO mapred.JobClient:     Reduce input groups=9284

09/02/06 18:13:23 INFO mapred.JobClient:     Combine output records=18568

09/02/06 18:13:23 INFO mapred.JobClient:     Map input records=7868

09/02/06 18:13:23 INFO mapred.JobClient:     Reduce output records=9284

09/02/06 18:13:23 INFO mapred.JobClient:     Map output bytes=445846

09/02/06 18:13:23 INFO mapred.JobClient:     Map input bytes=320950

09/02/06 18:13:23 INFO mapred.JobClient:     Combine input records=47227

09/02/06 18:13:23 INFO mapred.JobClient:     Map output records=37943

09/02/06 18:13:23 INFO mapred.JobClient:     Reduce input records=9284