Changes between Initial Version and Version 1 of waue/2011/chukwa

Timestamp:

Jan 28, 2011, 5:03:26 PM (13 years ago)

Author:

waue

Comment:

--

waue/2011/chukwa

@@ +1 @@
+
+hadoop 0.21.1
+chukwa 0.4
+ub1 是 jobtracker / namenode
+ub2 chukwa server / mysql server
+
+> The Chukwa hadoop cluster (CC)
+> The monitored source nodes (SN)
+> The monitored source nodes set up as a hadoop cluster (SN-C)
+
+保險起見，cc sn 都建立資料夾 
+/chukwa/
+/tmp/chukwa
+
+sudo apt-get install sysstat
+
+= cc 要設定的 conf/ =
+
+== alert ==
+waue@nchc.org.tw
+
+== chukwa-collector-conf.xml == 
+
+  <property>
+    <name>writer.hdfs.filesystem</name>
+    <value>hdfs://ub2:9000/</value>
+    <description>HDFS to dump to</description>
+  </property>
+  <property>
+    <name>chukwaCollector.outputDir</name>
+    <value>/chukwa/logs/</value>
+    <description>Chukwa data sink directory</description>
+  </property>
+  <property>
+    <name>chukwaCollector.http.port</name>
+    <value>8080</value>
+    <description>The HTTP port number the collector will listen on</description>
+  </property>
+
+== jdbc.conf ==
+ jdbc.conf.template 改成 jdbc.conf
+demo=jdbc:mysql://ub2:3306/test?user=root
+
+== nagios.properties ==
+log4j.appender.NAGIOS.Host=ub2
+ 
+== 預設值即可不用改 ==
+aggregator.sql 
+chukwa-demux-conf.xml 
+chukwa-log4j.properties 
+commons-logging.properties 
+database_create_tables.sql 
+log4j.properties mdl.xml
+
+
+= cc, sn 都要設定的 conf =
+
+== chukwa-env.sh == 
+export JAVA_HOME=/usr/lib/jvm/java-6-sun
+export HADOOP_HOME="/opt/hadoop"
+export HADOOP_CONF_DIR="/opt/hadoop/conf"
+
+= sn 要設定的 conf =
+
+== agents == 
+ agents.template ==>  agents
+
+ub1
+ub2
+
+== chukwa-agent-conf.xml == 
+chukwa-agent-conf.xml.template ==> chukwa-agent-conf.xml
+
+  <property>
+    <name>chukwaAgent.tags</name>
+    <value>cluster="wauegroup"</value>
+    <description>The cluster's name for this agent</description>
+  </property>
+
+  <property>
+    <name>chukwaAgent.hostname</name>
+    <value>localhost</value>
+    <description>The hostname of the agent on this node. Usually localhost, this is used by the chukwa instrumentation agent-control interface library</description>
+  </property>
+
+== collectors == 
+localhost
+
+== initial_adaptors == 
+cp initial_adaptors.template initial_adaptors
+
+ 
+
+= cc =
+安裝 mysql , php , phpmyadmin, apache2
+
+開 database : test
+匯入：  conf/database_create_tables.sql
+
+
+
+==  log4j.properties ==
+
+$ vim hadoop/conf/log4j.properties
+
+    log4j.appender.DRFA=org.apache.log4j.net.SocketAppender
+    log4j.appender.DRFA.RemoteHost=ub2
+    log4j.appender.DRFA.Port=9096
+    log4j.appender.DRFA.layout=org.apache.log4j.PatternLayout
+    log4j.appender.DRFA.layout.ConversionPattern=%d{ISO8601} %p %c: %m%n
+
+
+
+
+$ cd /opt/hadoop
+$ cp /opt/chukwa/conf/hadoop-metrics.properties.template conf/
+$ cp conf/hadoop-metrics.properties conf/hadoop-metrics
+$ cp /opt/chukwa/chukwa-hadoop-0.4.0-client.jar ./
+$ cp /opt/chukwa/chukwa-hadoop-0.4.0-client.jar ./lib/
+
+
+
+Configuring and starting the Collector
+
+   1. Copy conf/chukwa-collector-conf.xml.template to conf/chukwa-collector-conf.xml
+   2. Edit conf/chukwa-collector-conf.xml and comment out the default properties for chukwaCollector.writerClass, and chukwaCollector.pipeline. Uncomment block for HBaseWriter parameters, and save.
+   3. If you're running HBase in distributed mode, copy your hbase-site.xml file to the collectors conf/ directory. At a minimum, this file must contain a setting for hbase.zookeeper.quorum.
+   4. Copy conf/chukwa-env.sh-template to conf/chukwa-env.sh.
+   5. Edit chukwa-env.sh. You almost certainly need to set JAVA_HOME, HADOOP_HOME, HADOOP_CONF_DIR, HBASE_HOME, and HBASE_CONF_DIR at least.
+   6.
+
+      In the chukwa root directory, say bash bin/chukwa collector 
+
+Configuring and starting the local agent
+
+   1.
+
+      Copy conf/chukwa-agent-conf.xml.template to conf/chukwa-agent-conf.xml
+   2.
+
+      Copy conf/collectors.template to conf/collectors
+   3.
+
+      In the chukwa root directory, say bash bin/chukwa agent 
+
+Starting Adaptors
+
+The local agent speaks a simple text-based protocol, by default over port 9093. Suppose you want Chukwa to monitor system metrics, hadoop metrics, and hadoop logs on the localhost:
+
+   1. Telnet to localhost 9093
+   2.
+
+      Type [without quotation marks] "add org.apache.hadoop.chukwa.datacollection.adaptor.sigar.SystemMetrics SystemMetrics 60 0"
+   3.
+
+      Type [without quotation marks] "add SocketAdaptor HadoopMetrics 9095 0"
+   4.
+
+      Type [without quotation marks] "add SocketAdaptor Hadoop 9096 0"
+   5. Type "list" -- you should see the adaptor you just started, listed as running.
+   6. Type "close" to break the connection.
+   7.
+
+      If you don't have telnet, you can get the same effect using the netcat (nc) command line tool. 
+
+Set Up Cluster Aggregation Script
+
+For data analytics with pig, there are some additional environment setup. Pig does not use the same environment variable name as Hadoop, therefore make sure the following environment are setup correctly:
+
+   1. export PIG_CLASSPATH=$HADOOP_CONF_DIR:$HBASE_CONF_DIR
+   2. Setup a cron job for "pig -Dpig.additional.jars=${HBASE_HOME}/hbase-0.20.6.jar:${PIG_PATH}/pig.jar ${CHUKWA_HOME}/script/pig/ClusterSummary.pig" to run periodically 
+
+Set Up HICC
+
+The Hadoop Infrastructure Care Center (HICC) is the Chukwa web user interface. To set up HICC, do the following:
+
+   1. bin/chukwa hicc 
+
+Data visualization
+
+   1.
+
+      Point web browser to http://localhost:4080/hicc/jsp/graph_explorer.jsp
+   2. The default user name and password is "demo" without quotes.
+   3. System Metrics collected by Chukwa collector will be browsable through graph_explorer.jsp file.