Changes between Initial Version and Version 1 of III140412/Lab20

Timestamp:

Apr 11, 2014, 11:30:18 PM (12 years ago)

Author:

jazz

Comment:

--

III140412/Lab20

@@ +1 @@
+◢ <[wiki:III140412/Lab19 實作十九]> | <[wiki:III140412 回課程大綱]> ▲ | <[wiki:III140412/Lab21 實作二十一]> ◣
+
+= 實作二十 Lab20 =
+
+{{{
+#!html
+<p style="text-align: center;"><big style="font-weight: bold;"><big>預設的輸入格式<br/>TextInputFormat</big></big></p>
+}}}
+
+[[PageOutline]]
+
+{{{
+#!text
+請先連線至 nodeN.3du.me , N 為您的報名編號
+}}}
+
+ * 為了觀察 !FileInputFormat 的行為，我們使用 update jar 的技巧，對 !TextInputFormat.java 做了小幅度的修改。
+ * 官方實作的 !TextInputFormat.java 有兩個(一個新版,一個舊版)
+{{{
+lab@node1:~$ find /home/user/hadoop/src/ -name "TextInputFormat.java"
+/home/user/hadoop/src/mapred/org/apache/hadoop/mapreduce/lib/input/TextInputFormat.java
+/home/user/hadoop/src/mapred/org/apache/hadoop/mapred/TextInputFormat.java
+}}}
+ * 這裡我們修改新版的 !TextInputFormat.java
+{{{
+#!diff
+--- /home/user/hadoop/src/mapred/org/apache/hadoop/mapreduce/lib/input/TextInputFormat.java   2012-10-03 13:17:16.000000000 +0800
++++ /home/user/hadoop_labs/lab011/src/TextInputFormat.java      2013-10-19 11:25:16.419320587 +0800
+@@ -38,11 +38,13 @@
+   public RecordReader<LongWritable, Text>
+     createRecordReader(InputSplit split,
+                        TaskAttemptContext context) {
++    System.err.println("TextInputFormat.createRecordReader()");
+     return new LineRecordReader();
+   }
+
+   @Override
+   @Override
+   protected boolean isSplitable(JobContext context, Path file) {
++    System.err.println("TextInputFormat.isSplitable(context," + file.toString() + ")");
+     CompressionCodec codec =
+       new CompressionCodecFactory(context.getConfiguration()).getCodec(file);
+     return codec == null;
+}}}
+ * 讓我們先來觀察一下執行的結果
+{{{
+unset HADOOP_CONF_DIR
+cd ~/hadoop_labs/lab011
+ant
+cd ~/hadoop_labs/lab010
+mkdir -p my_input
+echo "A B C D" > my_input/input1
+echo "C D A B" > my_input/input2
+hadoop fs -put my_input my_input
+sed -i 's#setNumReduceTasks(0)#setNumReduceTasks(1)#g' ~/hadoop_labs/lab010/src/WordCount.java
+ant
+hadoop jar WordCount.jar my_input my_output
+}}}
+
+ * 
+{{{
+export HADOOP_CONF_DIR=~/hadoop/conf.local/
+hadoop jar WordCount.jar my_input my_output
+unset HADOOP_CONF_DIR
+}}}
+
+ * Reference:
+  1. http://hadoop.apache.org/docs/r1.0.4/api/org/apache/hadoop/mapreduce/InputFormat.html
+  2. http://hadoop.apache.org/docs/r1.0.4/api/org/apache/hadoop/mapreduce/lib/input/FileInputFormat.html
+  3. http://hadoop.apache.org/docs/r1.0.4/api/org/apache/hadoop/mapreduce/lib/input/TextInputFormat.html
+
+== 實作習題 ==
+
+ <問題 1> 當運行於全分散式模式，請問執行任務時，您觀察到幾行 "!TextInputFormat.isSplitable"
+
+ <問題 2> 當運行於全分散式模式，請問執行任務時，您觀察到幾行 "!TextInputFormat.createRecordReader"
+
+ <問題 3> 當運行於單機模式，請問執行任務時，您觀察到幾行 "!TextInputFormat.isSplitable"
+
+ <問題 4> 當運行於單機模式，請問執行任務時，您觀察到幾行 "!TextInputFormat.createRecordReader"