Uploaded image for project: 'Chukwa'
  1. Chukwa
  2. CHUKWA-563

Collector Crashes when HBase is not started

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 0.4.0
    • Fix Version/s: 0.5.0
    • Component/s: Data Collection
    • Labels:
      None
    • Environment:

      Ubuntu 8.04

      Description

      When using HBaseWriter, collectors crash if they start and do not find HBase running. Collectors also crash if HBase is running and Tables are not found

      Here are the exceptions in collector.log when HBase is not running

      2010-12-03 18:20:38,857 INFO main ChukwaConfiguration - chukwaConf is /home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../conf
      2010-12-03 18:20:39,188 INFO main root - initing servletCollector
      2010-12-03 18:20:39,190 INFO main PipelineStageWriter - using pipelined writers, pipe length is 2
      2010-12-03 18:20:39,198 INFO Thread-6 SocketTeeWriter - listen thread started
      2010-12-03 18:20:39,200 INFO main ChukwaConfiguration - chukwaConf is /home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../conf
      2010-12-03 18:20:39,396 INFO main ZooKeeper - Client environment:zookeeper.version=3.2.2-888565, built on 12/08/2009 21:51 GMT
      2010-12-03 18:20:39,396 INFO main ZooKeeper - Client environment:host.name=ahmed-laptop
      2010-12-03 18:20:39,397 INFO main ZooKeeper - Client environment:java.version=1.6.0_20
      2010-12-03 18:20:39,397 INFO main ZooKeeper - Client environment:java.vendor=Sun Microsystems Inc.
      2010-12-03 18:20:39,397 INFO main ZooKeeper - Client environment:java.home=/usr/lib/jvm/java-6-sun-1.6.0.20/jre
      2010-12-03 18:20:39,397 INFO main ZooKeeper - Client environment:java.class.path=/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../conf:/usr/local/hadoop/conf::/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../chukwa-core-0.5.0.jar:/usr/local/hadoop/hadoop-0.20.2-core.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/activemq-core-5.3.1.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/activemq-protobuf-1.0.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/aopalliance-1.0.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/asm-3.1.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/commons-beanutils-1.8.0.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/commons-cli-2.0-SNAPSHOT.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/commons-codec-1.3.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/commons-fileupload-1.2.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/commons-httpclient-3.1.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/commons-io-1.4.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/commons-lang-2.3.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/commons-logging-1.1.1.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/commons-logging-api-1.1.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/commons-net-2.0.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/confspellcheck.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/core-3.1.1.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/ezmorph-1.0.6.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/geronimo-j2ee-management_1.0_spec-1.0.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/geronimo-jms_1.1_spec-1.1.1.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/guava-r05.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/hadoop-core-0.20.2.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/hadoop-test-0.20.2.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/hbase-0.20.6.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/hbase-0.20.6-test.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/html-filter-1.0.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/jchronic-0.2.3.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/jersey-bundle-1.1.0-ea.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/jetty-6.1.11.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/jetty-util-6.1.11.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/json.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/json-lib-2.2.3-jdk15.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/jsp-2.1-6.1.11.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/jsp-api-2.1-6.1.11.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/jsr311-api-1.0.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/junit-3.8.1.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/kahadb-5.3.1.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/log4j-1.2.14.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/mysql-connector-java-5.1.6.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/NagiosAppender-1.5.0.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/org.osgi.core-4.1.0.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/prefuse.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/servlet-api-2.5-6.1.11.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/sigar.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/spring-beans-2.5.6.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/spring-context-2.5.6.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/spring-core-2.5.6.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/spring-mock-1.2.6.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/spring-osgi-core-1.2.1.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/spring-osgi-io-1.2.1.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/zookeeper-3.2.2.jar:
      2010-12-03 18:20:39,397 INFO main ZooKeeper - Client environment:java.library.path=/usr/local/hadoop/lib/native/Linux-i386-32
      2010-12-03 18:20:39,397 INFO main ZooKeeper - Client environment:java.io.tmpdir=/tmp
      2010-12-03 18:20:39,397 INFO main ZooKeeper - Client environment:java.compiler=<NA>
      2010-12-03 18:20:39,397 INFO main ZooKeeper - Client environment:os.name=Linux
      2010-12-03 18:20:39,397 INFO main ZooKeeper - Client environment:os.arch=i386
      2010-12-03 18:20:39,397 INFO main ZooKeeper - Client environment:os.version=2.6.24-19-generic
      2010-12-03 18:20:39,397 INFO main ZooKeeper - Client environment:user.name=hadoop
      2010-12-03 18:20:39,397 INFO main ZooKeeper - Client environment:user.home=/home/hadoop
      2010-12-03 18:20:39,397 INFO main ZooKeeper - Client environment:user.dir=/home/hadoop/workspace/chukwa-0.5.0-incubating
      2010-12-03 18:20:39,398 INFO main ZooKeeper - Initiating client connection, connectString=localhost:2181 sessionTimeout=60000 watcher=org.apache.hadoop.hbase.client.HConnectionManager$ClientZKWatcher@1428ea
      2010-12-03 18:20:39,400 INFO main ClientCnxn - zookeeper.disableAutoWatchReset is false
      2010-12-03 18:20:39,418 INFO main-SendThread ClientCnxn - Attempting connection to server localhost/127.0.0.1:2181
      2010-12-03 18:20:39,422 WARN main-SendThread ClientCnxn - Exception closing session 0x0 to sun.nio.ch.SelectionKeyImpl@1a68ef9
      java.net.ConnectException: Connection refused
      at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
      at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
      at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:933)
      2010-12-03 18:20:39,423 WARN main-SendThread ClientCnxn - Ignoring exception during shutdown input
      java.nio.channels.ClosedChannelException
      at sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:638)
      at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
      at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:999)
      at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
      2010-12-03 18:20:39,423 WARN main-SendThread ClientCnxn - Ignoring exception during shutdown output
      java.nio.channels.ClosedChannelException
      at sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:649)
      at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368)
      at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1004)
      at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
      2010-12-03 18:20:39,545 INFO main HConnectionManager$TableServers - getMaster attempt 0 of 10 failed; retrying after sleep of 1000
      java.io.IOException: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase/master
      at org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.readAddressOrThrow(ZooKeeperWrapper.java:394)
      at org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.readMasterAddressOrThrow(ZooKeeperWrapper.java:298)
      at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getMaster(HConnectionManager.java:339)
      at org.apache.hadoop.hbase.client.HBaseAdmin.<init>(HBaseAdmin.java:72)
      at org.apache.hadoop.chukwa.datacollection.writer.hbase.HBaseWriter.verifyHbaseSchema(HBaseWriter.java:146)
      at org.apache.hadoop.chukwa.datacollection.writer.hbase.HBaseWriter.init(HBaseWriter.java:116)
      at org.apache.hadoop.chukwa.datacollection.writer.PipelineStageWriter.init(PipelineStageWriter.java:88)
      at org.apache.hadoop.chukwa.datacollection.collector.servlet.ServletCollector.init(ServletCollector.java:112)
      at org.mortbay.jetty.servlet.ServletHolder.initServlet(ServletHolder.java:433)
      at org.mortbay.jetty.servlet.ServletHolder.doStart(ServletHolder.java:256)
      at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:39)
      at org.mortbay.jetty.servlet.ServletHandler.initialize(ServletHandler.java:616)
      at org.mortbay.jetty.servlet.Context.startContext(Context.java:140)
      at org.mortbay.jetty.handler.ContextHandler.doStart(ContextHandler.java:513)
      at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:39)
      at org.mortbay.jetty.handler.HandlerWrapper.doStart(HandlerWrapper.java:130)
      at org.mortbay.jetty.Server.doStart(Server.java:222)
      at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:39)
      at org.apache.hadoop.chukwa.datacollection.collector.CollectorStub.main(CollectorStub.java:136)
      Caused by: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase/master
      at org.apache.zookeeper.KeeperException.create(KeeperException.java:90)
      at org.apache.zookeeper.KeeperException.create(KeeperException.java:42)
      at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:892)
      at org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.readAddressOrThrow(ZooKeeperWrapper.java:390)
      ... 18 more
      2010-12-03 18:20:40,188 INFO Timer-1 root - stats:ServletCollector,numberHTTPConnection:0,numberchunks:0
      2010-12-03 18:20:40,312 INFO Timer-2 HBaseWriter - stat=HBaseWriter|dataRate=0
      2010-12-03 18:20:40,640 INFO main-SendThread ClientCnxn - Attempting connection to server localhost/127.0.0.1:2181
      2010-12-03 18:20:40,641 WARN main-SendThread ClientCnxn - Exception closing session 0x0 to sun.nio.ch.SelectionKeyImpl@1977b9b
      java.net.ConnectException: Connection refused
      at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
      at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
      at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:933)
      2010-12-03 18:20:40,641 WARN main-SendThread ClientCnxn - Ignoring exception during shutdown input
      java.nio.channels.ClosedChannelException
      at sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:638)
      at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
      at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:999)
      at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
      2010-12-03 18:20:40,641 WARN main-SendThread ClientCnxn - Ignoring exception during shutdown output
      java.nio.channels.ClosedChannelException
      at sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:649)
      at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368)
      at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1004)
      at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
      2010-12-03 18:20:40,742 INFO main HConnectionManager$TableServers - getMaster attempt 1 of 10 failed; retrying after sleep of 1000
      java.io.IOException: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase/master
      at org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.readAddressOrThrow(ZooKeeperWrapper.java:394)
      at org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.readMasterAddressOrThrow(ZooKeeperWrapper.java:298)
      at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getMaster(HConnectionManager.java:339)
      at org.apache.hadoop.hbase.client.HBaseAdmin.<init>(HBaseAdmin.java:72)
      at org.apache.hadoop.chukwa.datacollection.writer.hbase.HBaseWriter.verifyHbaseSchema(HBaseWriter.java:146)
      at org.apache.hadoop.chukwa.datacollection.writer.hbase.HBaseWriter.init(HBaseWriter.java:116)
      at org.apache.hadoop.chukwa.datacollection.writer.PipelineStageWriter.init(PipelineStageWriter.java:88)
      at org.apache.hadoop.chukwa.datacollection.collector.servlet.ServletCollector.init(ServletCollector.java:112)
      at org.mortbay.jetty.servlet.ServletHolder.initServlet(ServletHolder.java:433)
      at org.mortbay.jetty.servlet.ServletHolder.doStart(ServletHolder.java:256)
      at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:39)
      at org.mortbay.jetty.servlet.ServletHandler.initialize(ServletHandler.java:616)
      at org.mortbay.jetty.servlet.Context.startContext(Context.java:140)
      at org.mortbay.jetty.handler.ContextHandler.doStart(ContextHandler.java:513)
      at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:39)
      at org.mortbay.jetty.handler.HandlerWrapper.doStart(HandlerWrapper.java:130)
      at org.mortbay.jetty.Server.doStart(Server.java:222)
      at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:39)
      at org.apache.hadoop.chukwa.datacollection.collector.CollectorStub.main(CollectorStub.java:136)
      Caused by: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase/master
      at org.apache.zookeeper.KeeperException.create(KeeperException.java:90)
      at org.apache.zookeeper.KeeperException.create(KeeperException.java:42)
      at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:892)
      at org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.readAddressOrThrow(ZooKeeperWrapper.java:390)
      ... 18 more
      2010-12-03 18:20:42,329 INFO main-SendThread ClientCnxn - Attempting connection to server localhost/127.0.0.1:2181
      2010-12-03 18:20:42,330 WARN main-SendThread ClientCnxn - Exception closing session 0x0 to sun.nio.ch.SelectionKeyImpl@2e1f1f
      java.net.ConnectException: Connection refused
      at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
      at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
      at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:933)
      2010-12-03 18:20:42,330 WARN main-SendThread ClientCnxn - Ignoring exception during shutdown input
      java.nio.channels.ClosedChannelException
      at sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:638)
      at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
      at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:999)
      at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
      2010-12-03 18:20:42,330 WARN main-SendThread ClientCnxn - Ignoring exception during shutdown output
      java.nio.channels.ClosedChannelException
      at sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:649)
      at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368)
      at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1004)
      at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
      2010-12-03 18:20:42,431 INFO main HConnectionManager$TableServers - getMaster attempt 2 of 10 failed; retrying after sleep of 1000
      java.io.IOException: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase/master
      at org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.readAddressOrThrow(ZooKeeperWrapper.java:394)
      at org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.readMasterAddressOrThrow(ZooKeeperWrapper.java:298)
      at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getMaster(HConnectionManager.java:339)
      at org.apache.hadoop.hbase.client.HBaseAdmin.<init>(HBaseAdmin.java:72)
      at org.apache.hadoop.chukwa.datacollection.writer.hbase.HBaseWriter.verifyHbaseSchema(HBaseWriter.java:146)
      at org.apache.hadoop.chukwa.datacollection.writer.hbase.HBaseWriter.init(HBaseWriter.java:116)
      at org.apache.hadoop.chukwa.datacollection.writer.PipelineStageWriter.init(PipelineStageWriter.java:88)
      at org.apache.hadoop.chukwa.datacollection.collector.servlet.ServletCollector.init(ServletCollector.java:112)
      at org.mortbay.jetty.servlet.ServletHolder.initServlet(ServletHolder.java:433)
      at org.mortbay.jetty.servlet.ServletHolder.doStart(ServletHolder.java:256)
      at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:39)
      at org.mortbay.jetty.servlet.ServletHandler.initialize(ServletHandler.java:616)
      at org.mortbay.jetty.servlet.Context.startContext(Context.java:140)
      at org.mortbay.jetty.handler.ContextHandler.doStart(ContextHandler.java:513)
      at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:39)
      at org.mortbay.jetty.handler.HandlerWrapper.doStart(HandlerWrapper.java:130)
      at org.mortbay.jetty.Server.doStart(Server.java:222)
      at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:39)
      at org.apache.hadoop.chukwa.datacollection.collector.CollectorStub.main(CollectorStub.java:136)
      Caused by: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase/master
      at org.apache.zookeeper.KeeperException.create(KeeperException.java:90)
      at org.apache.zookeeper.KeeperException.create(KeeperException.java:42)
      at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:892)
      at org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.readAddressOrThrow(ZooKeeperWrapper.java:390)
      ... 18 more
      2010-12-03 18:20:43,903 INFO main-SendThread ClientCnxn - Attempting connection to server localhost/127.0.0.1:2181
      2010-12-03 18:20:43,904 WARN main-SendThread ClientCnxn - Exception closing session 0x0 to sun.nio.ch.SelectionKeyImpl@fefe3f
      java.net.ConnectException: Connection refused
      at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
      at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
      at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:933)
      2010-12-03 18:20:43,904 WARN main-SendThread ClientCnxn - Ignoring exception during shutdown input
      java.nio.channels.ClosedChannelException
      at sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:638)
      at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
      at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:999)
      at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
      2010-12-03 18:20:43,904 WARN main-SendThread ClientCnxn - Ignoring exception during shutdown output
      java.nio.channels.ClosedChannelException
      at sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:649)
      at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368)
      at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1004)
      at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
      2010-12-03 18:20:44,007 INFO main HConnectionManager$TableServers - getMaster attempt 3 of 10 failed; retrying after sleep of 2000
      java.io.IOException: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase/master
      at org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.readAddressOrThrow(ZooKeeperWrapper.java:394)
      at org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.readMasterAddressOrThrow(ZooKeeperWrapper.java:298)
      at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getMaster(HConnectionManager.java:339)
      at org.apache.hadoop.hbase.client.HBaseAdmin.<init>(HBaseAdmin.java:72)
      at org.apache.hadoop.chukwa.datacollection.writer.hbase.HBaseWriter.verifyHbaseSchema(HBaseWriter.java:146)
      at org.apache.hadoop.chukwa.datacollection.writer.hbase.HBaseWriter.init(HBaseWriter.java:116)
      at org.apache.hadoop.chukwa.datacollection.writer.PipelineStageWriter.init(PipelineStageWriter.java:88)
      at org.apache.hadoop.chukwa.datacollection.collector.servlet.ServletCollector.init(ServletCollector.java:112)
      at org.mortbay.jetty.servlet.ServletHolder.initServlet(ServletHolder.java:433)
      at org.mortbay.jetty.servlet.ServletHolder.doStart(ServletHolder.java:256)
      at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:39)
      at org.mortbay.jetty.servlet.ServletHandler.initialize(ServletHandler.java:616)
      at org.mortbay.jetty.servlet.Context.startContext(Context.java:140)
      at org.mortbay.jetty.handler.ContextHandler.doStart(ContextHandler.java:513)
      at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:39)
      at org.mortbay.jetty.handler.HandlerWrapper.doStart(HandlerWrapper.java:130)
      at org.mortbay.jetty.Server.doStart(Server.java:222)
      at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:39)
      at org.apache.hadoop.chukwa.datacollection.collector.CollectorStub.main(CollectorStub.java:136)
      Caused by: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase/master
      at org.apache.zookeeper.KeeperException.create(KeeperException.java:90)
      at org.apache.zookeeper.KeeperException.create(KeeperException.java:42)
      at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:892)
      at org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.readAddressOrThrow(ZooKeeperWrapper.java:390)
      ... 18 more
      2010-12-03 18:20:45,178 INFO main-SendThread ClientCnxn - Attempting connection to server localhost/127.0.0.1:2181
      2010-12-03 18:20:45,178 WARN main-SendThread ClientCnxn - Exception closing session 0x0 to sun.nio.ch.SelectionKeyImpl@c2b2f6
      java.net.ConnectException: Connection refused
      at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
      at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
      at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:933)
      2010-12-03 18:20:45,179 WARN main-SendThread ClientCnxn - Ignoring exception during shutdown input
      java.nio.channels.ClosedChannelException
      at sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:638)
      at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
      at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:999)
      at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
      2010-12-03 18:20:45,179 WARN main-SendThread ClientCnxn - Ignoring exception during shutdown output
      java.nio.channels.ClosedChannelException
      at sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:649)
      at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368)
      at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1004)
      at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
      2010-12-03 18:20:46,610 INFO main-SendThread ClientCnxn - Attempting connection to server localhost/127.0.0.1:2181
      2010-12-03 18:20:46,610 WARN main-SendThread ClientCnxn - Exception closing session 0x0 to sun.nio.ch.SelectionKeyImpl@1a001ff
      java.net.ConnectException: Connection refused
      at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
      at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
      at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:933)
      2010-12-03 18:20:46,611 WARN main-SendThread ClientCnxn - Ignoring exception during shutdown input
      java.nio.channels.ClosedChannelException
      at sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:638)
      at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
      at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:999)
      at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
      2010-12-03 18:20:46,611 WARN main-SendThread ClientCnxn - Ignoring exception during shutdown output
      java.nio.channels.ClosedChannelException
      at sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:649)
      at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368)
      at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1004)
      at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
      2010-12-03 18:20:46,711 INFO main HConnectionManager$TableServers - getMaster attempt 4 of 10 failed; retrying after sleep of 2000
      java.io.IOException: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase/master
      at org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.readAddressOrThrow(ZooKeeperWrapper.java:394)
      at org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.readMasterAddressOrThrow(ZooKeeperWrapper.java:298)
      at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getMaster(HConnectionManager.java:339)
      at org.apache.hadoop.hbase.client.HBaseAdmin.<init>(HBaseAdmin.java:72)
      at org.apache.hadoop.chukwa.datacollection.writer.hbase.HBaseWriter.verifyHbaseSchema(HBaseWriter.java:146)
      at org.apache.hadoop.chukwa.datacollection.writer.hbase.HBaseWriter.init(HBaseWriter.java:116)
      at org.apache.hadoop.chukwa.datacollection.writer.PipelineStageWriter.init(PipelineStageWriter.java:88)
      at org.apache.hadoop.chukwa.datacollection.collector.servlet.ServletCollector.init(ServletCollector.java:112)
      at org.mortbay.jetty.servlet.ServletHolder.initServlet(ServletHolder.java:433)
      at org.mortbay.jetty.servlet.ServletHolder.doStart(ServletHolder.java:256)
      at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:39)
      at org.mortbay.jetty.servlet.ServletHandler.initialize(ServletHandler.java:616)
      at org.mortbay.jetty.servlet.Context.startContext(Context.java:140)
      at org.mortbay.jetty.handler.ContextHandler.doStart(ContextHandler.java:513)
      at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:39)
      at org.mortbay.jetty.handler.HandlerWrapper.doStart(HandlerWrapper.java:130)
      at org.mortbay.jetty.Server.doStart(Server.java:222)
      at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:39)
      at org.apache.hadoop.chukwa.datacollection.collector.CollectorStub.main(CollectorStub.java:136)
      Caused by: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase/master
      at org.apache.zookeeper.KeeperException.create(KeeperException.java:90)
      at org.apache.zookeeper.KeeperException.create(KeeperException.java:42)
      at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:892)
      at org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.readAddressOrThrow(ZooKeeperWrapper.java:390)
      ... 18 more
      2010-12-03 18:20:48,252 INFO main-SendThread ClientCnxn - Attempting connection to server localhost/127.0.0.1:2181
      2010-12-03 18:20:48,252 WARN main-SendThread ClientCnxn - Exception closing session 0x0 to sun.nio.ch.SelectionKeyImpl@90c06f
      java.net.ConnectException: Connection refused
      at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
      at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
      at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:933)
      2010-12-03 18:20:48,253 WARN main-SendThread ClientCnxn - Ignoring exception during shutdown input
      java.nio.channels.ClosedChannelException
      at sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:638)
      at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
      at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:999)
      at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
      2010-12-03 18:20:48,253 WARN main-SendThread ClientCnxn - Ignoring exception during shutdown output
      java.nio.channels.ClosedChannelException
      at sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:649)
      at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368)
      at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1004)
      at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
      2010-12-03 18:20:49,973 INFO main-SendThread ClientCnxn - Attempting connection to server localhost/127.0.0.1:2181
      2010-12-03 18:20:49,973 WARN main-SendThread ClientCnxn - Exception closing session 0x0 to sun.nio.ch.SelectionKeyImpl@1267649
      java.net.ConnectException: Connection refused
      at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
      at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
      at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:933)
      2010-12-03 18:20:49,973 WARN main-SendThread ClientCnxn - Ignoring exception during shutdown input
      java.nio.channels.ClosedChannelException
      at sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:638)
      at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
      at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:999)
      at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
      2010-12-03 18:20:49,974 WARN main-SendThread ClientCnxn - Ignoring exception during shutdown output
      java.nio.channels.ClosedChannelException
      at sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:649)
      at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368)
      at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1004)
      at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
      2010-12-03 18:20:50,074 INFO main HConnectionManager$TableServers - getMaster attempt 5 of 10 failed; retrying after sleep of 4000
      java.io.IOException: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase/master
      at org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.readAddressOrThrow(ZooKeeperWrapper.java:394)
      at org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.readMasterAddressOrThrow(ZooKeeperWrapper.java:298)
      at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getMaster(HConnectionManager.java:339)
      at org.apache.hadoop.hbase.client.HBaseAdmin.<init>(HBaseAdmin.java:72)
      at org.apache.hadoop.chukwa.datacollection.writer.hbase.HBaseWriter.verifyHbaseSchema(HBaseWriter.java:146)
      at org.apache.hadoop.chukwa.datacollection.writer.hbase.HBaseWriter.init(HBaseWriter.java:116)
      at org.apache.hadoop.chukwa.datacollection.writer.PipelineStageWriter.init(PipelineStageWriter.java:88)
      at org.apache.hadoop.chukwa.datacollection.collector.servlet.ServletCollector.init(ServletCollector.java:112)
      at org.mortbay.jetty.servlet.ServletHolder.initServlet(ServletHolder.java:433)
      at org.mortbay.jetty.servlet.ServletHolder.doStart(ServletHolder.java:256)
      at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:39)
      at org.mortbay.jetty.servlet.ServletHandler.initialize(ServletHandler.java:616)
      at org.mortbay.jetty.servlet.Context.startContext(Context.java:140)
      at org.mortbay.jetty.handler.ContextHandler.doStart(ContextHandler.java:513)
      at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:39)
      at org.mortbay.jetty.handler.HandlerWrapper.doStart(HandlerWrapper.java:130)
      at org.mortbay.jetty.Server.doStart(Server.java:222)
      at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:39)
      at org.apache.hadoop.chukwa.datacollection.collector.CollectorStub.main(CollectorStub.java:136)
      Caused by: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase/master
      at org.apache.zookeeper.KeeperException.create(KeeperException.java:90)
      at org.apache.zookeeper.KeeperException.create(KeeperException.java:42)
      at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:892)
      at org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.readAddressOrThrow(ZooKeeperWrapper.java:390)
      ... 18 more
      2010-12-03 18:20:50,312 INFO Timer-2 HBaseWriter - stat=HBaseWriter|dataRate=0
      2010-12-03 18:20:51,749 INFO main-SendThread ClientCnxn - Attempting connection to server localhost/127.0.0.1:2181
      2010-12-03 18:20:51,750 WARN main-SendThread ClientCnxn - Exception closing session 0x0 to sun.nio.ch.SelectionKeyImpl@dcb03b
      java.net.ConnectException: Connection refused
      at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
      at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
      at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:933)
      2010-12-03 18:20:51,750 WARN main-SendThread ClientCnxn - Ignoring exception during shutdown input
      java.nio.channels.ClosedChannelException
      at sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:638)
      at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
      at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:999)
      at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
      2010-12-03 18:20:51,750 WARN main-SendThread ClientCnxn - Ignoring exception during shutdown output
      java.nio.channels.ClosedChannelException
      at sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:649)
      at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368)
      at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1004)
      at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
      2010-12-03 18:20:53,639 INFO main-SendThread ClientCnxn - Attempting connection to server localhost/127.0.0.1:2181
      2010-12-03 18:20:53,640 WARN main-SendThread ClientCnxn - Exception closing session 0x0 to sun.nio.ch.SelectionKeyImpl@1b1fbf4
      java.net.ConnectException: Connection refused
      at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
      at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
      at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:933)
      2010-12-03 18:20:53,640 WARN main-SendThread ClientCnxn - Ignoring exception during shutdown input
      java.nio.channels.ClosedChannelException
      at sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:638)
      at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
      at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:999)
      at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
      2010-12-03 18:20:53,640 WARN main-SendThread ClientCnxn - Ignoring exception during shutdown output
      java.nio.channels.ClosedChannelException
      at sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:649)
      at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368)
      at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1004)
      at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
      2010-12-03 18:20:55,094 INFO main-SendThread ClientCnxn - Attempting connection to server localhost/127.0.0.1:2181
      2010-12-03 18:20:55,094 WARN main-SendThread ClientCnxn - Exception closing session 0x0 to sun.nio.ch.SelectionKeyImpl@1829e6f
      java.net.ConnectException: Connection refused
      at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
      at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
      at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:933)
      2010-12-03 18:20:55,094 WARN main-SendThread ClientCnxn - Ignoring exception during shutdown input
      java.nio.channels.ClosedChannelException
      at sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:638)
      at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
      at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:999)
      at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
      2010-12-03 18:20:55,095 WARN main-SendThread ClientCnxn - Ignoring exception during shutdown output
      java.nio.channels.ClosedChannelException
      at sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:649)
      at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368)
      at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1004)
      at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
      2010-12-03 18:20:55,195 INFO main HConnectionManager$TableServers - getMaster attempt 6 of 10 failed; retrying after sleep of 4000
      java.io.IOException: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase/master
      at org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.readAddressOrThrow(ZooKeeperWrapper.java:394)
      at org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.readMasterAddressOrThrow(ZooKeeperWrapper.java:298)
      at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getMaster(HConnectionManager.java:339)
      at org.apache.hadoop.hbase.client.HBaseAdmin.<init>(HBaseAdmin.java:72)
      at org.apache.hadoop.chukwa.datacollection.writer.hbase.HBaseWriter.verifyHbaseSchema(HBaseWriter.java:146)
      at org.apache.hadoop.chukwa.datacollection.writer.hbase.HBaseWriter.init(HBaseWriter.java:116)
      at org.apache.hadoop.chukwa.datacollection.writer.PipelineStageWriter.init(PipelineStageWriter.java:88)
      at org.apache.hadoop.chukwa.datacollection.collector.servlet.ServletCollector.init(ServletCollector.java:112)
      at org.mortbay.jetty.servlet.ServletHolder.initServlet(ServletHolder.java:433)
      at org.mortbay.jetty.servlet.ServletHolder.doStart(ServletHolder.java:256)
      at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:39)
      at org.mortbay.jetty.servlet.ServletHandler.initialize(ServletHandler.java:616)
      at org.mortbay.jetty.servlet.Context.startContext(Context.java:140)
      at org.mortbay.jetty.handler.ContextHandler.doStart(ContextHandler.java:513)
      at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:39)
      at org.mortbay.jetty.handler.HandlerWrapper.doStart(HandlerWrapper.java:130)
      at org.mortbay.jetty.Server.doStart(Server.java:222)
      at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:39)
      at org.apache.hadoop.chukwa.datacollection.collector.CollectorStub.main(CollectorStub.java:136)
      Caused by: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase/master
      at org.apache.zookeeper.KeeperException.create(KeeperException.java:90)
      at org.apache.zookeeper.KeeperException.create(KeeperException.java:42)
      at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:892)
      at org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.readAddressOrThrow(ZooKeeperWrapper.java:390)
      ... 18 more
      2010-12-03 18:20:56,869 INFO main-SendThread ClientCnxn - Attempting connection to server localhost/127.0.0.1:2181
      2010-12-03 18:20:56,869 WARN main-SendThread ClientCnxn - Exception closing session 0x0 to sun.nio.ch.SelectionKeyImpl@100ebec
      java.net.ConnectException: Connection refused
      at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
      at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
      at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:933)
      2010-12-03 18:20:56,869 WARN main-SendThread ClientCnxn - Ignoring exception during shutdown input
      java.nio.channels.ClosedChannelException
      at sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:638)
      at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
      at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:999)
      at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
      2010-12-03 18:20:56,870 WARN main-SendThread ClientCnxn - Ignoring exception during shutdown output
      java.nio.channels.ClosedChannelException
      at sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:649)
      at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368)
      at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1004)
      at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
      2010-12-03 18:20:58,099 INFO main-SendThread ClientCnxn - Attempting connection to server localhost/127.0.0.1:2181
      2010-12-03 18:20:58,100 WARN main-SendThread ClientCnxn - Exception closing session 0x0 to sun.nio.ch.SelectionKeyImpl@7736bd
      java.net.ConnectException: Connection refused
      at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
      at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
      at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:933)
      2010-12-03 18:20:58,100 WARN main-SendThread ClientCnxn - Ignoring exception during shutdown input
      java.nio.channels.ClosedChannelException
      at sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:638)
      at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
      at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:999)
      at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
      2010-12-03 18:20:58,100 WARN main-SendThread ClientCnxn - Ignoring exception during shutdown output
      java.nio.channels.ClosedChannelException
      at sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:649)
      at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368)
      at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1004)
      at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
      2010-12-03 18:20:59,363 INFO main-SendThread ClientCnxn - Attempting connection to server localhost/127.0.0.1:2181
      2010-12-03 18:20:59,364 WARN main-SendThread ClientCnxn - Exception closing session 0x0 to sun.nio.ch.SelectionKeyImpl@178460d
      java.net.ConnectException: Connection refused
      at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
      at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
      at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:933)
      2010-12-03 18:20:59,364 WARN main-SendThread ClientCnxn - Ignoring exception during shutdown input
      java.nio.channels.ClosedChannelException
      at sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:638)
      at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
      at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:999)
      at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
      2010-12-03 18:20:59,364 WARN main-SendThread ClientCnxn - Ignoring exception during shutdown output
      java.nio.channels.ClosedChannelException
      at sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:649)
      at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368)
      at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1004)
      at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
      2010-12-03 18:20:59,465 INFO main HConnectionManager$TableServers - getMaster attempt 7 of 10 failed; retrying after sleep of 8000
      java.io.IOException: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase/master
      at org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.readAddressOrThrow(ZooKeeperWrapper.java:394)
      at org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.readMasterAddressOrThrow(ZooKeeperWrapper.java:298)
      at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getMaster(HConnectionManager.java:339)
      at org.apache.hadoop.hbase.client.HBaseAdmin.<init>(HBaseAdmin.java:72)
      at org.apache.hadoop.chukwa.datacollection.writer.hbase.HBaseWriter.verifyHbaseSchema(HBaseWriter.java:146)
      at org.apache.hadoop.chukwa.datacollection.writer.hbase.HBaseWriter.init(HBaseWriter.java:116)
      at org.apache.hadoop.chukwa.datacollection.writer.PipelineStageWriter.init(PipelineStageWriter.java:88)
      at org.apache.hadoop.chukwa.datacollection.collector.servlet.ServletCollector.init(ServletCollector.java:112)
      at org.mortbay.jetty.servlet.ServletHolder.initServlet(ServletHolder.java:433)
      at org.mortbay.jetty.servlet.ServletHolder.doStart(ServletHolder.java:256)
      at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:39)
      at org.mortbay.jetty.servlet.ServletHandler.initialize(ServletHandler.java:616)
      at org.mortbay.jetty.servlet.Context.startContext(Context.java:140)
      at org.mortbay.jetty.handler.ContextHandler.doStart(ContextHandler.java:513)
      at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:39)
      at org.mortbay.jetty.handler.HandlerWrapper.doStart(HandlerWrapper.java:130)
      at org.mortbay.jetty.Server.doStart(Server.java:222)
      at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:39)
      at org.apache.hadoop.chukwa.datacollection.collector.CollectorStub.main(CollectorStub.java:136)
      Caused by: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase/master
      at org.apache.zookeeper.KeeperException.create(KeeperException.java:90)
      at org.apache.zookeeper.KeeperException.create(KeeperException.java:42)
      at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:892)
      at org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.readAddressOrThrow(ZooKeeperWrapper.java:390)
      ... 18 more
      2010-12-03 18:21:00,313 INFO Timer-2 HBaseWriter - stat=HBaseWriter|dataRate=0
      2010-12-03 18:21:00,703 INFO main-SendThread ClientCnxn - Attempting connection to server localhost/127.0.0.1:2181
      2010-12-03 18:21:00,704 WARN main-SendThread ClientCnxn - Exception closing session 0x0 to sun.nio.ch.SelectionKeyImpl@1938039
      java.net.ConnectException: Connection refused
      at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
      at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
      at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:933)
      2010-12-03 18:21:00,704 WARN main-SendThread ClientCnxn - Ignoring exception during shutdown input
      java.nio.channels.ClosedChannelException
      at sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:638)
      at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
      at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:999)
      at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
      2010-12-03 18:21:00,704 WARN main-SendThread ClientCnxn - Ignoring exception during shutdown output
      java.nio.channels.ClosedChannelException
      at sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:649)
      at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368)
      at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1004)
      at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
      2010-12-03 18:21:02,221 INFO main-SendThread ClientCnxn - Attempting connection to server localhost/127.0.0.1:2181
      2010-12-03 18:21:02,222 WARN main-SendThread ClientCnxn - Exception closing session 0x0 to sun.nio.ch.SelectionKeyImpl@3c342b
      java.net.ConnectException: Connection refused
      at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
      at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
      at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:933)
      2010-12-03 18:21:02,222 WARN main-SendThread ClientCnxn - Ignoring exception during shutdown input
      java.nio.channels.ClosedChannelException
      at sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:638)
      at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
      at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:999)
      at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
      2010-12-03 18:21:02,222 WARN main-SendThread ClientCnxn - Ignoring exception during shutdown output
      java.nio.channels.ClosedChannelException
      at sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:649)
      at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368)
      at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1004)
      at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
      2010-12-03 18:21:03,664 INFO main-SendThread ClientCnxn - Attempting connection to server localhost/127.0.0.1:2181
      2010-12-03 18:21:03,665 WARN main-SendThread ClientCnxn - Exception closing session 0x0 to sun.nio.ch.SelectionKeyImpl@419d05
      java.net.ConnectException: Connection refused
      at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
      at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
      at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:933)
      2010-12-03 18:21:03,665 WARN main-SendThread ClientCnxn - Ignoring exception during shutdown input
      java.nio.channels.ClosedChannelException
      at sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:638)
      at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
      at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:999)
      at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
      2010-12-03 18:21:03,665 WARN main-SendThread ClientCnxn - Ignoring exception during shutdown output
      java.nio.channels.ClosedChannelException
      at sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:649)
      at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368)
      at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1004)
      at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
      2010-12-03 18:21:04,922 INFO main-SendThread ClientCnxn - Attempting connection to server localhost/127.0.0.1:2181
      2010-12-03 18:21:04,922 WARN main-SendThread ClientCnxn - Exception closing session 0x0 to sun.nio.ch.SelectionKeyImpl@1deeb40
      java.net.ConnectException: Connection refused
      at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
      at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
      at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:933)
      2010-12-03 18:21:04,923 WARN main-SendThread ClientCnxn - Ignoring exception during shutdown input
      java.nio.channels.ClosedChannelException
      at sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:638)
      at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
      at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:999)
      at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
      2010-12-03 18:21:04,923 WARN main-SendThread ClientCnxn - Ignoring exception during shutdown output
      java.nio.channels.ClosedChannelException
      at sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:649)
      at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368)
      at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1004)
      at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
      2010-12-03 18:21:06,779 INFO main-SendThread ClientCnxn - Attempting connection to server localhost/127.0.0.1:2181
      2010-12-03 18:21:06,779 WARN main-SendThread ClientCnxn - Exception closing session 0x0 to sun.nio.ch.SelectionKeyImpl@cafb56
      java.net.ConnectException: Connection refused
      at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
      at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
      at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:933)
      2010-12-03 18:21:06,779 WARN main-SendThread ClientCnxn - Ignoring exception during shutdown input
      java.nio.channels.ClosedChannelException
      at sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:638)
      at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
      at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:999)
      at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
      2010-12-03 18:21:06,780 WARN main-SendThread ClientCnxn - Ignoring exception during shutdown output
      java.nio.channels.ClosedChannelException
      at sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:649)
      at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368)
      at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1004)
      at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
      2010-12-03 18:21:07,942 INFO main-SendThread ClientCnxn - Attempting connection to server localhost/127.0.0.1:2181
      2010-12-03 18:21:07,943 WARN main-SendThread ClientCnxn - Exception closing session 0x0 to sun.nio.ch.SelectionKeyImpl@1a7ddcf
      java.net.ConnectException: Connection refused
      at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
      at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
      at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:933)
      2010-12-03 18:21:07,944 WARN main-SendThread ClientCnxn - Ignoring exception during shutdown input
      java.nio.channels.ClosedChannelException
      at sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:638)
      at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
      at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:999)
      at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
      2010-12-03 18:21:07,944 WARN main-SendThread ClientCnxn - Ignoring exception during shutdown output
      java.nio.channels.ClosedChannelException
      at sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:649)
      at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368)
      at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1004)
      at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
      2010-12-03 18:21:08,044 INFO main HConnectionManager$TableServers - getMaster attempt 8 of 10 failed; retrying after sleep of 16000
      java.io.IOException: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase/master
      at org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.readAddressOrThrow(ZooKeeperWrapper.java:394)
      at org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.readMasterAddressOrThrow(ZooKeeperWrapper.java:298)
      at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getMaster(HConnectionManager.java:339)
      at org.apache.hadoop.hbase.client.HBaseAdmin.<init>(HBaseAdmin.java:72)
      at org.apache.hadoop.chukwa.datacollection.writer.hbase.HBaseWriter.verifyHbaseSchema(HBaseWriter.java:146)
      at org.apache.hadoop.chukwa.datacollection.writer.hbase.HBaseWriter.init(HBaseWriter.java:116)
      at org.apache.hadoop.chukwa.datacollection.writer.PipelineStageWriter.init(PipelineStageWriter.java:88)
      at org.apache.hadoop.chukwa.datacollection.collector.servlet.ServletCollector.init(ServletCollector.java:112)
      at org.mortbay.jetty.servlet.ServletHolder.initServlet(ServletHolder.java:433)
      at org.mortbay.jetty.servlet.ServletHolder.doStart(ServletHolder.java:256)
      at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:39)
      at org.mortbay.jetty.servlet.ServletHandler.initialize(ServletHandler.java:616)
      at org.mortbay.jetty.servlet.Context.startContext(Context.java:140)
      at org.mortbay.jetty.handler.ContextHandler.doStart(ContextHandler.java:513)
      at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:39)
      at org.mortbay.jetty.handler.HandlerWrapper.doStart(HandlerWrapper.java:130)
      at org.mortbay.jetty.Server.doStart(Server.java:222)
      at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:39)
      at org.apache.hadoop.chukwa.datacollection.collector.CollectorStub.main(CollectorStub.java:136)
      Caused by: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase/master
      at org.apache.zookeeper.KeeperException.create(KeeperException.java:90)
      at org.apache.zookeeper.KeeperException.create(KeeperException.java:42)
      at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:892)
      at org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.readAddressOrThrow(ZooKeeperWrapper.java:390)
      ... 18 more
      2010-12-03 18:21:09,964 INFO main-SendThread ClientCnxn - Attempting connection to server localhost/127.0.0.1:2181
      2010-12-03 18:21:09,965 WARN main-SendThread ClientCnxn - Exception closing session 0x0 to sun.nio.ch.SelectionKeyImpl@1da669c
      java.net.ConnectException: Connection refused
      at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
      at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
      at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:933)
      2010-12-03 18:21:09,965 WARN main-SendThread ClientCnxn - Ignoring exception during shutdown input
      java.nio.channels.ClosedChannelException
      at sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:638)
      at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
      at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:999)
      at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
      2010-12-03 18:21:09,966 WARN main-SendThread ClientCnxn - Ignoring exception during shutdown output
      java.nio.channels.ClosedChannelException
      at sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:649)
      at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368)
      at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1004)
      at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
      2010-12-03 18:21:10,313 INFO Timer-2 HBaseWriter - stat=HBaseWriter|dataRate=0
      2010-12-03 18:21:11,285 INFO main-SendThread ClientCnxn - Attempting connection to server localhost/127.0.0.1:2181
      2010-12-03 18:21:11,286 WARN main-SendThread ClientCnxn - Exception closing session 0x0 to sun.nio.ch.SelectionKeyImpl@17f242c
      java.net.ConnectException: Connection refused
      at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
      at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
      at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:933)
      2010-12-03 18:21:11,286 WARN main-SendThread ClientCnxn - Ignoring exception during shutdown input
      java.nio.channels.ClosedChannelException
      at sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:638)
      at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
      at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:999)
      at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
      2010-12-03 18:21:11,286 WARN main-SendThread ClientCnxn - Ignoring exception during shutdown output
      java.nio.channels.ClosedChannelException
      at sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:649)
      at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368)
      at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1004)
      at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
      2010-12-03 18:21:12,919 INFO main-SendThread ClientCnxn - Attempting connection to server localhost/127.0.0.1:2181
      2010-12-03 18:21:12,920 WARN main-SendThread ClientCnxn - Exception closing session 0x0 to sun.nio.ch.SelectionKeyImpl@12f9ee
      java.net.ConnectException: Connection refused
      at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
      at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
      at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:933)
      2010-12-03 18:21:12,920 WARN main-SendThread ClientCnxn - Ignoring exception during shutdown input
      java.nio.channels.ClosedChannelException
      at sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:638)
      at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
      at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:999)
      at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
      2010-12-03 18:21:12,921 WARN main-SendThread ClientCnxn - Ignoring exception during shutdown output
      java.nio.channels.ClosedChannelException
      at sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:649)
      at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368)
      at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1004)
      at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
      2010-12-03 18:21:14,754 INFO main-SendThread ClientCnxn - Attempting connection to server localhost/127.0.0.1:2181
      2010-12-03 18:21:14,755 WARN main-SendThread ClientCnxn - Exception closing session 0x0 to sun.nio.ch.SelectionKeyImpl@106fc94
      java.net.ConnectException: Connection refused
      at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
      at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
      at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:933)
      2010-12-03 18:21:14,755 WARN main-SendThread ClientCnxn - Ignoring exception during shutdown input
      java.nio.channels.ClosedChannelException
      at sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:638)
      at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
      at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:999)
      at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
      2010-12-03 18:21:14,755 WARN main-SendThread ClientCnxn - Ignoring exception during shutdown output
      java.nio.channels.ClosedChannelException
      at sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:649)
      at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368)
      at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1004)
      at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
      2010-12-03 18:21:15,947 INFO main-SendThread ClientCnxn - Attempting connection to server localhost/127.0.0.1:2181
      2010-12-03 18:21:15,948 WARN main-SendThread ClientCnxn - Exception closing session 0x0 to sun.nio.ch.SelectionKeyImpl@7f5580
      java.net.ConnectException: Connection refused
      at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
      at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
      at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:933)
      2010-12-03 18:21:15,948 WARN main-SendThread ClientCnxn - Ignoring exception during shutdown input
      java.nio.channels.ClosedChannelException
      at sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:638)
      at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
      at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:999)
      at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
      2010-12-03 18:21:15,948 WARN main-SendThread ClientCnxn - Ignoring exception during shutdown output
      java.nio.channels.ClosedChannelException
      at sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:649)
      at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368)
      at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1004)
      at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
      2010-12-03 18:21:17,483 INFO main-SendThread ClientCnxn - Attempting connection to server localhost/127.0.0.1:2181
      2010-12-03 18:21:17,484 WARN main-SendThread ClientCnxn - Exception closing session 0x0 to sun.nio.ch.SelectionKeyImpl@84cc09
      java.net.ConnectException: Connection refused
      at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
      at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
      at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:933)
      2010-12-03 18:21:17,484 WARN main-SendThread ClientCnxn - Ignoring exception during shutdown input
      java.nio.channels.ClosedChannelException
      at sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:638)
      at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
      at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:999)
      at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
      2010-12-03 18:21:17,484 WARN main-SendThread ClientCnxn - Ignoring exception during shutdown output
      java.nio.channels.ClosedChannelException
      at sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:649)
      at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368)
      at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1004)
      at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
      2010-12-03 18:21:18,793 INFO main-SendThread ClientCnxn - Attempting connection to server localhost/127.0.0.1:2181
      2010-12-03 18:21:18,794 WARN main-SendThread ClientCnxn - Exception closing session 0x0 to sun.nio.ch.SelectionKeyImpl@14db0e3
      java.net.ConnectException: Connection refused
      at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
      at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
      at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:933)
      2010-12-03 18:21:18,794 WARN main-SendThread ClientCnxn - Ignoring exception during shutdown input
      java.nio.channels.ClosedChannelException
      at sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:638)
      at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
      at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:999)
      at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
      2010-12-03 18:21:18,794 WARN main-SendThread ClientCnxn - Ignoring exception during shutdown output
      java.nio.channels.ClosedChannelException
      at sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:649)
      at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368)
      at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1004)
      at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
      2010-12-03 18:21:19,964 INFO main-SendThread ClientCnxn - Attempting connection to server localhost/127.0.0.1:2181
      2010-12-03 18:21:19,965 WARN main-SendThread ClientCnxn - Exception closing session 0x0 to sun.nio.ch.SelectionKeyImpl@676437
      java.net.ConnectException: Connection refused
      at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
      at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
      at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:933)
      2010-12-03 18:21:19,965 WARN main-SendThread ClientCnxn - Ignoring exception during shutdown input
      java.nio.channels.ClosedChannelException
      at sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:638)
      at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
      at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:999)
      at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
      2010-12-03 18:21:19,965 WARN main-SendThread ClientCnxn - Ignoring exception during shutdown output
      java.nio.channels.ClosedChannelException
      at sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:649)
      at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368)
      at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1004)
      at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
      2010-12-03 18:21:20,313 INFO Timer-2 HBaseWriter - stat=HBaseWriter|dataRate=0
      2010-12-03 18:21:21,899 INFO main-SendThread ClientCnxn - Attempting connection to server localhost/127.0.0.1:2181
      2010-12-03 18:21:21,900 WARN main-SendThread ClientCnxn - Exception closing session 0x0 to sun.nio.ch.SelectionKeyImpl@1e808ca
      java.net.ConnectException: Connection refused
      at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
      at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
      at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:933)
      2010-12-03 18:21:21,900 WARN main-SendThread ClientCnxn - Ignoring exception during shutdown input
      java.nio.channels.ClosedChannelException
      at sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:638)
      at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
      at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:999)
      at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
      2010-12-03 18:21:21,900 WARN main-SendThread ClientCnxn - Ignoring exception during shutdown output
      java.nio.channels.ClosedChannelException
      at sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:649)
      at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368)
      at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1004)
      at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
      2010-12-03 18:21:23,916 INFO main-SendThread ClientCnxn - Attempting connection to server localhost/127.0.0.1:2181
      2010-12-03 18:21:23,917 WARN main-SendThread ClientCnxn - Exception closing session 0x0 to sun.nio.ch.SelectionKeyImpl@2bd3a
      java.net.ConnectException: Connection refused
      at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
      at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
      at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:933)
      2010-12-03 18:21:23,917 WARN main-SendThread ClientCnxn - Ignoring exception during shutdown input
      java.nio.channels.ClosedChannelException
      at sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:638)
      at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
      at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:999)
      at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
      2010-12-03 18:21:23,917 WARN main-SendThread ClientCnxn - Ignoring exception during shutdown output
      java.nio.channels.ClosedChannelException
      at sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:649)
      at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368)
      at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1004)
      at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
      2010-12-03 18:21:25,857 INFO main-SendThread ClientCnxn - Attempting connection to server localhost/127.0.0.1:2181
      2010-12-03 18:21:25,857 WARN main-SendThread ClientCnxn - Exception closing session 0x0 to sun.nio.ch.SelectionKeyImpl@1329642
      java.net.ConnectException: Connection refused
      at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
      at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
      at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:933)
      2010-12-03 18:21:25,858 WARN main-SendThread ClientCnxn - Ignoring exception during shutdown input
      java.nio.channels.ClosedChannelException
      at sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:638)
      at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
      at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:999)
      at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
      2010-12-03 18:21:25,858 WARN main-SendThread ClientCnxn - Ignoring exception during shutdown output
      java.nio.channels.ClosedChannelException
      at sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:649)
      at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368)
      at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1004)
      at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970)
      2010-12-03 18:21:25,958 INFO main HConnectionManager$TableServers - getMaster attempt 9 of 10 failed; no more retrying.
      java.io.IOException: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase/master
      at org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.readAddressOrThrow(ZooKeeperWrapper.java:394)
      at org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.readMasterAddressOrThrow(ZooKeeperWrapper.java:298)
      at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getMaster(HConnectionManager.java:339)
      at org.apache.hadoop.hbase.client.HBaseAdmin.<init>(HBaseAdmin.java:72)
      at org.apache.hadoop.chukwa.datacollection.writer.hbase.HBaseWriter.verifyHbaseSchema(HBaseWriter.java:146)
      at org.apache.hadoop.chukwa.datacollection.writer.hbase.HBaseWriter.init(HBaseWriter.java:116)
      at org.apache.hadoop.chukwa.datacollection.writer.PipelineStageWriter.init(PipelineStageWriter.java:88)
      at org.apache.hadoop.chukwa.datacollection.collector.servlet.ServletCollector.init(ServletCollector.java:112)
      at org.mortbay.jetty.servlet.ServletHolder.initServlet(ServletHolder.java:433)
      at org.mortbay.jetty.servlet.ServletHolder.doStart(ServletHolder.java:256)
      at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:39)
      at org.mortbay.jetty.servlet.ServletHandler.initialize(ServletHandler.java:616)
      at org.mortbay.jetty.servlet.Context.startContext(Context.java:140)
      at org.mortbay.jetty.handler.ContextHandler.doStart(ContextHandler.java:513)
      at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:39)
      at org.mortbay.jetty.handler.HandlerWrapper.doStart(HandlerWrapper.java:130)
      at org.mortbay.jetty.Server.doStart(Server.java:222)
      at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:39)
      at org.apache.hadoop.chukwa.datacollection.collector.CollectorStub.main(CollectorStub.java:136)
      Caused by: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase/master
      at org.apache.zookeeper.KeeperException.create(KeeperException.java:90)
      at org.apache.zookeeper.KeeperException.create(KeeperException.java:42)
      at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:892)
      at org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.readAddressOrThrow(ZooKeeperWrapper.java:390)
      ... 18 more
      2010-12-03 18:21:25,961 ERROR main HBaseWriter - org.apache.hadoop.hbase.MasterNotRunningException
      at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getMaster(HConnectionManager.java:374)
      at org.apache.hadoop.hbase.client.HBaseAdmin.<init>(HBaseAdmin.java:72)
      at org.apache.hadoop.chukwa.datacollection.writer.hbase.HBaseWriter.verifyHbaseSchema(HBaseWriter.java:146)
      at org.apache.hadoop.chukwa.datacollection.writer.hbase.HBaseWriter.init(HBaseWriter.java:116)
      at org.apache.hadoop.chukwa.datacollection.writer.PipelineStageWriter.init(PipelineStageWriter.java:88)
      at org.apache.hadoop.chukwa.datacollection.collector.servlet.ServletCollector.init(ServletCollector.java:112)
      at org.mortbay.jetty.servlet.ServletHolder.initServlet(ServletHolder.java:433)
      at org.mortbay.jetty.servlet.ServletHolder.doStart(ServletHolder.java:256)
      at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:39)
      at org.mortbay.jetty.servlet.ServletHandler.initialize(ServletHandler.java:616)
      at org.mortbay.jetty.servlet.Context.startContext(Context.java:140)
      at org.mortbay.jetty.handler.ContextHandler.doStart(ContextHandler.java:513)
      at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:39)
      at org.mortbay.jetty.handler.HandlerWrapper.doStart(HandlerWrapper.java:130)
      at org.mortbay.jetty.Server.doStart(Server.java:222)
      at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:39)
      at org.apache.hadoop.chukwa.datacollection.collector.CollectorStub.main(CollectorStub.java:136)

      2010-12-03 18:21:25,961 ERROR main HBaseWriter - Hbase schema mismatch with demux parser.
      2010-12-03 18:21:25,961 ERROR main HBaseWriter - Exiting...
      2010-12-03 18:21:25,963 INFO HCM.shutdownHook ZooKeeper - Closing session: 0x0
      2010-12-03 18:21:25,964 INFO HCM.shutdownHook ClientCnxn - Closing ClientCnxn for session: 0x0
      2010-12-03 18:21:27,098 INFO main-SendThread ClientCnxn - Attempting connection to server localhost/127.0.0.1:2181
      2010-12-03 18:21:27,099 INFO main-SendThread ClientCnxn - Exception while closing send thread for session 0x0 : Connection refused
      2010-12-03 18:21:27,099 WARN main-SendThread ClientCnxn - Ignoring exception during shutdown input
      java.nio.channels.ClosedChannelException
      at sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:638)
      at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
      at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:999)
      at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:984)
      2010-12-03 18:21:27,099 WARN main-SendThread ClientCnxn - Ignoring exception during shutdown output
      java.nio.channels.ClosedChannelException
      at sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:649)
      at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368)
      at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1004)
      at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:984)
      2010-12-03 18:21:27,199 INFO HCM.shutdownHook ClientCnxn - Disconnecting ClientCnxn for session: 0x0
      2010-12-03 18:21:27,200 INFO HCM.shutdownHook ZooKeeper - Session: 0x0 closed
      2010-12-03 18:21:27,200 INFO main-EventThread ClientCnxn - EventThread shut down

      Here are the exceptions when HBase is running but tables do not exist

      010-12-03 18:24:40,846 INFO main ChukwaConfiguration - chukwaConf is /home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../conf
      2010-12-03 18:24:41,230 INFO main root - initing servletCollector
      2010-12-03 18:24:41,232 INFO main PipelineStageWriter - using pipelined writers, pipe length is 2
      2010-12-03 18:24:41,235 INFO Thread-6 SocketTeeWriter - listen thread started
      2010-12-03 18:24:41,238 INFO main ChukwaConfiguration - chukwaConf is /home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../conf
      2010-12-03 18:24:41,418 INFO main ZooKeeper - Client environment:zookeeper.version=3.2.2-888565, built on 12/08/2009 21:51 GMT
      2010-12-03 18:24:41,418 INFO main ZooKeeper - Client environment:host.name=ahmed-laptop
      2010-12-03 18:24:41,418 INFO main ZooKeeper - Client environment:java.version=1.6.0_20
      2010-12-03 18:24:41,418 INFO main ZooKeeper - Client environment:java.vendor=Sun Microsystems Inc.
      2010-12-03 18:24:41,418 INFO main ZooKeeper - Client environment:java.home=/usr/lib/jvm/java-6-sun-1.6.0.20/jre
      2010-12-03 18:24:41,418 INFO main ZooKeeper - Client environment:java.class.path=/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../conf:/usr/local/hadoop/conf::/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../chukwa-core-0.5.0.jar:/usr/local/hadoop/hadoop-0.20.2-core.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/activemq-core-5.3.1.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/activemq-protobuf-1.0.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/aopalliance-1.0.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/asm-3.1.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/commons-beanutils-1.8.0.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/commons-cli-2.0-SNAPSHOT.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/commons-codec-1.3.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/commons-fileupload-1.2.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/commons-httpclient-3.1.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/commons-io-1.4.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/commons-lang-2.3.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/commons-logging-1.1.1.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/commons-logging-api-1.1.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/commons-net-2.0.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/confspellcheck.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/core-3.1.1.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/ezmorph-1.0.6.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/geronimo-j2ee-management_1.0_spec-1.0.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/geronimo-jms_1.1_spec-1.1.1.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/guava-r05.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/hadoop-core-0.20.2.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/hadoop-test-0.20.2.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/hbase-0.20.6.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/hbase-0.20.6-test.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/html-filter-1.0.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/jchronic-0.2.3.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/jersey-bundle-1.1.0-ea.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/jetty-6.1.11.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/jetty-util-6.1.11.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/json.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/json-lib-2.2.3-jdk15.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/jsp-2.1-6.1.11.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/jsp-api-2.1-6.1.11.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/jsr311-api-1.0.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/junit-3.8.1.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/kahadb-5.3.1.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/log4j-1.2.14.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/mysql-connector-java-5.1.6.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/NagiosAppender-1.5.0.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/org.osgi.core-4.1.0.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/prefuse.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/servlet-api-2.5-6.1.11.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/sigar.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/spring-beans-2.5.6.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/spring-context-2.5.6.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/spring-core-2.5.6.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/spring-mock-1.2.6.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/spring-osgi-core-1.2.1.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/spring-osgi-io-1.2.1.jar:/home/hadoop/workspace/chukwa-0.5.0-incubating/bin/../lib/zookeeper-3.2.2.jar:
      2010-12-03 18:24:41,418 INFO main ZooKeeper - Client environment:java.library.path=/usr/local/hadoop/lib/native/Linux-i386-32
      2010-12-03 18:24:41,419 INFO main ZooKeeper - Client environment:java.io.tmpdir=/tmp
      2010-12-03 18:24:41,419 INFO main ZooKeeper - Client environment:java.compiler=<NA>
      2010-12-03 18:24:41,419 INFO main ZooKeeper - Client environment:os.name=Linux
      2010-12-03 18:24:41,419 INFO main ZooKeeper - Client environment:os.arch=i386
      2010-12-03 18:24:41,419 INFO main ZooKeeper - Client environment:os.version=2.6.24-19-generic
      2010-12-03 18:24:41,419 INFO main ZooKeeper - Client environment:user.name=hadoop
      2010-12-03 18:24:41,419 INFO main ZooKeeper - Client environment:user.home=/home/hadoop
      2010-12-03 18:24:41,419 INFO main ZooKeeper - Client environment:user.dir=/home/hadoop/workspace/chukwa-0.5.0-incubating
      2010-12-03 18:24:41,420 INFO main ZooKeeper - Initiating client connection, connectString=localhost:2181 sessionTimeout=60000 watcher=org.apache.hadoop.hbase.client.HConnectionManager$ClientZKWatcher@1326484
      2010-12-03 18:24:41,422 INFO main ClientCnxn - zookeeper.disableAutoWatchReset is false
      2010-12-03 18:24:41,440 INFO main-SendThread ClientCnxn - Attempting connection to server localhost/127.0.0.1:2181
      2010-12-03 18:24:41,442 INFO main-SendThread ClientCnxn - Priming connection to java.nio.channels.SocketChannel[connected local=/127.0.0.1:58057 remote=localhost/127.0.0.1:2181]
      2010-12-03 18:24:41,500 INFO main-SendThread ClientCnxn - Server connection successful
      2010-12-03 18:24:41,984 ERROR main HBaseWriter - java.lang.Exception: HBase table: Hadoop does not exist.
      at org.apache.hadoop.chukwa.datacollection.writer.hbase.HBaseWriter.verifyHbaseTable(HBaseWriter.java:133)
      at org.apache.hadoop.chukwa.datacollection.writer.hbase.HBaseWriter.verifyHbaseSchema(HBaseWriter.java:158)
      at org.apache.hadoop.chukwa.datacollection.writer.hbase.HBaseWriter.init(HBaseWriter.java:116)
      at org.apache.hadoop.chukwa.datacollection.writer.PipelineStageWriter.init(PipelineStageWriter.java:88)
      at org.apache.hadoop.chukwa.datacollection.collector.servlet.ServletCollector.init(ServletCollector.java:112)
      at org.mortbay.jetty.servlet.ServletHolder.initServlet(ServletHolder.java:433)
      at org.mortbay.jetty.servlet.ServletHolder.doStart(ServletHolder.java:256)
      at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:39)
      at org.mortbay.jetty.servlet.ServletHandler.initialize(ServletHandler.java:616)
      at org.mortbay.jetty.servlet.Context.startContext(Context.java:140)
      at org.mortbay.jetty.handler.ContextHandler.doStart(ContextHandler.java:513)
      at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:39)
      at org.mortbay.jetty.handler.HandlerWrapper.doStart(HandlerWrapper.java:130)
      at org.mortbay.jetty.Server.doStart(Server.java:222)
      at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:39)
      at org.apache.hadoop.chukwa.datacollection.collector.CollectorStub.main(CollectorStub.java:136)

      2010-12-03 18:24:41,984 ERROR main HBaseWriter - java.lang.Exception: Validation failed - table: Hadoop column family: ClientTrace does not exist.
      at org.apache.hadoop.chukwa.datacollection.writer.hbase.HBaseWriter.verifyHbaseSchema(HBaseWriter.java:159)
      at org.apache.hadoop.chukwa.datacollection.writer.hbase.HBaseWriter.init(HBaseWriter.java:116)
      at org.apache.hadoop.chukwa.datacollection.writer.PipelineStageWriter.init(PipelineStageWriter.java:88)
      at org.apache.hadoop.chukwa.datacollection.collector.servlet.ServletCollector.init(ServletCollector.java:112)
      at org.mortbay.jetty.servlet.ServletHolder.initServlet(ServletHolder.java:433)
      at org.mortbay.jetty.servlet.ServletHolder.doStart(ServletHolder.java:256)
      at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:39)
      at org.mortbay.jetty.servlet.ServletHandler.initialize(ServletHandler.java:616)
      at org.mortbay.jetty.servlet.Context.startContext(Context.java:140)
      at org.mortbay.jetty.handler.ContextHandler.doStart(ContextHandler.java:513)
      at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:39)
      at org.mortbay.jetty.handler.HandlerWrapper.doStart(HandlerWrapper.java:130)
      at org.mortbay.jetty.Server.doStart(Server.java:222)
      at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:39)
      at org.apache.hadoop.chukwa.datacollection.collector.CollectorStub.main(CollectorStub.java:136)

      2010-12-03 18:24:41,984 ERROR main HBaseWriter - Hbase schema mismatch with demux parser.
      2010-12-03 18:24:41,984 ERROR main HBaseWriter - Exiting...
      2010-12-03 18:24:42,087 INFO HCM.shutdownHook ZooKeeper - Closing session: 0x12cad0c07fd0002
      2010-12-03 18:24:42,087 INFO HCM.shutdownHook ClientCnxn - Closing ClientCnxn for session: 0x12cad0c07fd0002
      2010-12-03 18:24:42,094 INFO main-SendThread ClientCnxn - Exception while closing send thread for session 0x12cad0c07fd0002 : Read error rc = -1 java.nio.DirectByteBuffer[pos=0 lim=4 cap=4]
      2010-12-03 18:24:42,202 INFO HCM.shutdownHook ClientCnxn - Disconnecting ClientCnxn for session: 0x12cad0c07fd0002
      2010-12-03 18:24:42,202 INFO HCM.shutdownHook ZooKeeper - Session: 0x12cad0c07fd0002 closed
      2010-12-03 18:24:42,202 INFO main-EventThread ClientCnxn - EventThread shut down
      2010-12-03 18:24:42,231 INFO Timer-1 root - stats:ServletCollector,numberHTTPConnection:0,numberchunks:0

        Activity

        Hide
        eyang Eric Yang added a comment -

        HBaseWriter runs a schema validation to make sure there is no misconfiguration. The schema validation is currently mandatory. If the validation does not pass, collector will bail out. We can add optional parameter to turn off schema validation.

        Show
        eyang Eric Yang added a comment - HBaseWriter runs a schema validation to make sure there is no misconfiguration. The schema validation is currently mandatory. If the validation does not pass, collector will bail out. We can add optional parameter to turn off schema validation.
        Hide
        billgraham Bill Graham added a comment -

        FWIW, this behavior is consistent with what happens when the collector starts and HDFS is down. The collector should be able to handle intermittent lack of connectivity with it's data source, but it seems reasonable to require the data source to be available for the collector to start up properly.

        Show
        billgraham Bill Graham added a comment - FWIW, this behavior is consistent with what happens when the collector starts and HDFS is down. The collector should be able to handle intermittent lack of connectivity with it's data source, but it seems reasonable to require the data source to be available for the collector to start up properly.
        Hide
        eyang Eric Yang added a comment -

        Added option:

        • hbase.writer.verify.schema - Verify HBase Table schema with demux parser schema, log warning if there are mismatch between hbase schema and demux parsers.
        • hbase.writer.halt.on.schema.mismatch - If this option is set to true, and HBase table schema is mismatched with demux parser, collector will shut down itself.
        Show
        eyang Eric Yang added a comment - Added option: hbase.writer.verify.schema - Verify HBase Table schema with demux parser schema, log warning if there are mismatch between hbase schema and demux parsers. hbase.writer.halt.on.schema.mismatch - If this option is set to true, and HBase table schema is mismatched with demux parser, collector will shut down itself.
        Hide
        eyang Eric Yang added a comment -

        The default setting is no verification applied.

        Show
        eyang Eric Yang added a comment - The default setting is no verification applied.
        Hide
        asrabkin Ari Rabkin added a comment -

        I vote +1 to commit this.

        Show
        asrabkin Ari Rabkin added a comment - I vote +1 to commit this.
        Hide
        eyang Eric Yang added a comment -

        Thanks Ari, I just committed this.

        Show
        eyang Eric Yang added a comment - Thanks Ari, I just committed this.

          People

          • Assignee:
            eyang Eric Yang
            Reporter:
            afathalla Ahmed Fathalla
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development