Changes between Version 27 and Version 28 of MR_manual

Timestamp:

Sep 3, 2008, 1:17:30 PM (18 years ago)

Author:

waue

Comment:

--

MR_manual

@@ -40 +40 @@
 [http://tech.ccidnet.com/art/5833/20080318/1393525_1.html copied from 詳細講解HBase]
  = 二、環境設定 = 
-
- == 2.1 Prepare == 
-System : 
- * Ubuntu 7.10
- * Hadoop 0.16
- * Hbase 0.1.3
-ps : hbase 0.1.4 <--> hadoop 0.2.0 
-Requirement : 
- * Eclipse   (3.2.2)
-{{{
-$ apt-get install eclipse
-}}}
-java 6  
-{{{
-$ apt-get install sun-java6-bin sun-java6-jdk sun-java6-jre sun-java6-plugin
-}}}
-suggest to remove the default java compiler 「 gcj 」
-{{{
-$ apt-get purge java-gcj-compat
-}}}
-Append two codes to /etc/bash.bashrc to setup Java Class path
-{{{
-export JAVA_HOME=/usr/lib/jvm/java-6-sun
-export HADOOP_HOME=/home/waue/workspace/hadoop/
-export CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar
-}}}
-Building UP Path
+所對應到的路徑為
  || Name || Path || 
  || Java Home || /usr/lib/jvm/java-6-sun || 
@@ -72 +46 @@
  || Hbase Home || /home/waue/workspace/hbase/ || 
 
-Nodes set
+節點
  || node name || server || 
  || cloud1 || v || 
  || cloud2 ||  ||
  || cloudn ||  ||
-
+ == 2.1 準備 == 
+系統 : 
+ * Ubuntu 7.10
+ * Hadoop 0.16
+ * Hbase 0.1.3
+ps : 若要升級則需要兩者都升級 hbase 0.1.4 <--> hadoop 0.2.0  
+ * Eclipse   (3.2.2)
+{{{
+$ apt-get install eclipse
+}}}
+java 6  
+{{{
+$ apt-get install sun-java6-bin sun-java6-jdk sun-java6-jre sun-java6-plugin
+}}}
+建議刪除原本的 「 gcj 」
+{{{
+$ apt-get purge java-gcj-compat
+}}}
+加入以下內容到 /etc/bash.bashrc
+{{{
+export JAVA_HOME=/usr/lib/jvm/java-6-sun
+export HADOOP_HOME=/home/waue/workspace/hadoop/
+export HBASE_HOME=/home/waue/workspace/hbase/
+export CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar
+}}}
  == 2.2 Hadoop Setup == 
  === 2.2.1. Generate an SSH key for the user === 
@@ -83 +81 @@
 $ ssh-keygen -t rsa -P ""
 $ cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
-$ ssh localhost
+$ ssh cloud1
 $ exit
 }}}
@@ -89 +87 @@
 {{{
  $ cd /home/waue/workspace
- $ sudo tar xzf hadoop-0.16.0.tar.gz
- $ sudo mv hadoop-0.16.0 hadoop
+ $ sudo tar xzf hadoop-0.16.3.tar.gz
+ $ sudo ln -sf hadoop-0.16.3 hadoop
  $ sudo chown -R waue:waue hadoop
  $ cd hadoop
@@ -106 +104 @@
 export JAVA_HOME=/usr/lib/jvm/java-6-sun
 export HADOOP_HOME=/home/waue/workspace/hadoop
+export HBASE_HOME=/home/waue/workspace/hbase
 export HADOOP_LOG_DIR=$HADOOP_HOME/logs
 export HADOOP_SLAVES=$HADOOP_HOME/conf/slaves
-}}}
+export HADOOP_CLASSPATH= $HBASE_HOME/hbase-0.1.3.jar:$HBASE_HOME/conf
+}}}
+ps. HADOOP_CLASSPATH 要設hbase 的環境，而HBASE_CLASSPATH要設hadoop的環境，
+有了這行可以解決編譯hbase 程式時出現run time error
+
  2. hadoop-site.xml  ($HADOOP_HOME/conf/)[[BR]]
 modify the contents of conf/hadoop-site.xml as below
@@ -127 +130 @@
 <property>
   <name>mapred.map.tasks</name>
-  <value>1</value>
+  <value>9</value>
   <description>
     define mapred.map tasks to be number of slave hosts
@@ -134 +137 @@
 <property>
   <name>mapred.reduce.tasks</name>
-  <value>1</value>
+  <value>9</value>
   <description>
     define mapred.reduce tasks to be number of slave hosts
@@ -175 +178 @@
 }}}
     * 用hbase連接hadoop DFS
-      * 編輯 conf/hbase-site.xml檔案,如下
-
+      * 編輯 conf/hbase-site.xml檔案如下，並複製一份到$HADOOP_HOME/conf 下
 {{{
 <configuration>
   <property>
       <name>hbase.master</name>
-      <value>localhost:60000</value>
+      <value>cloud1:60000</value>
   </property>
   <property>
       <name>hbase.master.info.bindAddress</name>
-      <value>localhost</value>
+      <value>cloud1</value>
       <description>The address for the hbase master web UI</description>
   </property>
 <property>
    <name>hbase.regionserver.info.bindAddress</name>
-   <value>localhost</value>
+   <value>cloud1</value>
    <description>The address for the hbase regionserver web UI
   </description>
@@ -197 +199 @@
      <name>hbase.rootdir</name>
      <value>file:///tmp/hbase-${user.home}/hbase</value>
-     <value>hdfs://localhost:9000/hbase</value>
+     <value>hdfs://cloud1:9000/hbase</value>
      <description>
          The directory shared by region servers.
@@ -206 +208 @@
 
 }}}
-      * 編輯 conf/hbase-site.xml檔案,如下
+      * 多nodes模式，編輯 conf/regionservers檔案,如下
 {{{
 cloud1
@@ -249 +251 @@
 {{{
 starting namenode, logging to /home/waue/workspace/hadoop/logs/hadoop-waue-namenode-Dx7200.out
-localhost: starting datanode, logging to /home/waue/workspace/hadoop/logs/hadoop-waue-datanode-Dx7200.out
-localhost: starting secondarynamenode, logging to /home/waue/workspace/hadoop/logs/hadoop-waue-secondarynamenode-Dx7200.out
+cloud1: starting datanode, logging to /home/waue/workspace/hadoop/logs/hadoop-waue-datanode-Dx7200.out
+cloud1: starting secondarynamenode, logging to /home/waue/workspace/hadoop/logs/hadoop-waue-secondarynamenode-Dx7200.out
 starting jobtracker, logging to /home/waue/workspace/hadoop/logs/hadoop-waue-jobtracker-Dx7200.out
-localhost: starting tasktracker, logging to /home/waue/workspace/hadoop/logs/hadoop-waue-tasktracker-Dx7200.out
-}}}
- * Then make sure http://localhost:50030/ by your explorer is on going.  [[br]]
+cloud1: starting tasktracker, logging to /home/waue/workspace/hadoop/logs/hadoop-waue-tasktracker-Dx7200.out
+}}}
+ * Then make sure http://cloud1:50030/ by your explorer is on going.  [[br]]
 
  * Ps : if your system had error after restart, you could do there for resolving and renewing one. And repeat to 「4. start up Hadoop」
@@ -381 +383 @@
 Click blue elephant to add a new MapReduce server location. 
 Server name : any_you_want 
-Hostname : localhost 
+Hostname : cloud1 
 Installation directory: /home/waue/workspace/nutch/ 
 Username : waue
@@ -412 +414 @@
  * A 「console」 tag will show beside 「!MapReduce Server」 tag. 
 
- * While Map Reduce is running, you can visit http://localhost:50030/ to view that Hadoop is dispatching jobs by Map Reduce.
-
- * After finish, you can go to http://localhost:50060/ to see the result.
+ * While Map Reduce is running, you can visit http://cloud1:50030/ to view that Hadoop is dispatching jobs by Map Reduce.
+
+ * After finish, you can go to http://cloud1:50060/ to see the result.
 
 
@@ -422 +424 @@
 
  = 七、Reference = 
-