◢ <[wiki:Hinet131022/Lab19 實作十九]> | <[wiki:Hinet131022 回課程大綱]> ▲ | <[wiki:Hinet131022/Lab21 實作二十一]> ◣ = 實作二十 Lab20 = {{{ #!html

預設的輸入格式
TextInputFormat

}}} [[PageOutline]] {{{ #!text 請先連線至 nodeN.3du.me , N 為您的報名編號 }}} * 為了觀察 !FileInputFormat 的行為,我們使用 update jar 的技巧,對 !TextInputFormat.java 做了小幅度的修改。 * 官方實作的 !TextInputFormat.java 有兩個(一個新版,一個舊版) {{{ user@node1:~$ find /home/user/hadoop/src/ -name "TextInputFormat.java" /home/user/hadoop/src/mapred/org/apache/hadoop/mapreduce/lib/input/TextInputFormat.java /home/user/hadoop/src/mapred/org/apache/hadoop/mapred/TextInputFormat.java }}} * 這裡我們修改新版的 !TextInputFormat.java {{{ #!diff --- /home/user/hadoop/src/mapred/org/apache/hadoop/mapreduce/lib/input/TextInputFormat.java 2012-10-03 13:17:16.000000000 +0800 +++ /home/user/hadoop_labs/lab011/src/TextInputFormat.java 2013-10-19 11:25:16.419320587 +0800 @@ -38,11 +38,13 @@ public RecordReader createRecordReader(InputSplit split, TaskAttemptContext context) { + System.err.println("TextInputFormat.createRecordReader()"); return new LineRecordReader(); } @Override @Override protected boolean isSplitable(JobContext context, Path file) { + System.err.println("TextInputFormat.isSplitable(context," + file.toString() + ")"); CompressionCodec codec = new CompressionCodecFactory(context.getConfiguration()).getCodec(file); return codec == null; }}} * 讓我們先來觀察一下執行的結果 {{{ unset HADOOP_CONF_DIR cd ~/hadoop_labs/lab011 ant cd ~/hadoop_labs/lab010 mkdir -p my_input echo "A B C D" > my_input/input1 echo "C D A B" > my_input/input2 hadoop fs -put my_input my_input sed -i 's#setNumReduceTasks(0)#setNumReduceTasks(1)#g' ~/hadoop_labs/lab010/src/WordCount.java ant hadoop jar WordCount.jar my_input my_output }}} * {{{ export HADOOP_CONF_DIR=~/hadoop/conf.local/ hadoop jar WordCount.jar my_input my_output unset HADOOP_CONF_DIR }}} * Reference: 1. http://hadoop.apache.org/docs/r1.0.4/api/org/apache/hadoop/mapreduce/InputFormat.html 2. http://hadoop.apache.org/docs/r1.0.4/api/org/apache/hadoop/mapreduce/lib/input/FileInputFormat.html 3. http://hadoop.apache.org/docs/r1.0.4/api/org/apache/hadoop/mapreduce/lib/input/TextInputFormat.html == 實作習題 == <問題 1> 當運行於全分散式模式,請問執行任務時,您觀察到幾行 "!TextInputFormat.isSplitable" <問題 2> 當運行於全分散式模式,請問執行任務時,您觀察到幾行 "!TextInputFormat.createRecordReader" <問題 3> 當運行於單機模式,請問執行任務時,您觀察到幾行 "!TextInputFormat.isSplitable" <問題 4> 當運行於單機模式,請問執行任務時,您觀察到幾行 "!TextInputFormat.createRecordReader"