wiki:HadoopWorkshopHandsOn

Version 3 (modified by jazz, 16 years ago) (diff)

--

Hadoop Hands-on Labs (1)

Basic DFS command / Hadoop DFS 基本測試環境建立

  1. download hadoop-0.18.2
    $ cd ~
    $ wget http://ftp.twaren.net/Unix/Web/apache/hadoop/core/hadoop-0.18.2/hadoop-0.18.2.tar.gz
    $ tar zxvf hadoop-0.18.2.tar.gz
    
  2. Hadoop 會用 SSH 進行內部連線,因此需要做 SSH Key exchange
    ~$ ssh-keygen
    ~$ cp ~/.ssh/id_rsa.pub ~/.ssh/authorized_keys
    
  3. 需要 JAVA_HOME 環境變數才能執行 hadoop namenode
    $ echo "export JAVA_HOME=/usr/lib/jvm/java-6-sun" >> ~/.bash_profile
    $ cd ~/hadoop-0.18.2
    
  4. 編輯 conf/hadoop-evn.sh (HADOOP_HOME要設定到你的hadoop安裝目錄)
    export JAVA_HOME=/usr/lib/jvm/java-6-sun
    export HADOOP_HOME=/home/jazz/hadoop-0.18.2/
    export HADOOP_CONF_DIR=$HADOOP_HOME/conf
    
  5. 編輯 conf/hadoop-site.xml 在 configuration 那一段加入以下設定
    <property>
      <name>fs.default.name</name>
      <value>hdfs://localhost:9000/</value>
      <description>
        The name of the default file system. Either the literal string
        "local" or a host:port for NDFS.
      </description>
    </property>
    <property>
      <name>mapred.job.tracker</name>
      <value>localhost:9001</value>
      <description>
        The host and port that the MapReduce job tracker runs at. If
        "local", then jobs are run in-process as a single map and
        reduce task.
      </description>
    </property>
    
  6. 啟動hadoop 的兩道指令
    ~/hadoop-0.18.2$ bin/hadoop namenode -format
    ~/hadoop-0.18.2$ bin/start-all.sh
    
  7. 完成後可以看到以下三個網頁
  1. 也可以放的東西上hdfs去看看
    ~/hadoop-0.18.2$ bin/hadoop dfs -put conf conf
    ~/hadoop-0.18.2$ bin/hadoop dfs -ls
    Found 1 items
    drwxr-xr-x   - jazz supergroup          0 2008-11-04 15:56 /user/jazz/conf