Mon Mar 30 07:55:17 UTC 2009 Starting regionserver on aa0-000-12.u.powerset.com ulimit -n 10240 2009-03-30 07:55:18,123 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: vmInputArguments=[-Xmx1000m, -XX:+HeapDumpOnOutOfMemoryError, -Dhbase.log.dir=/home/nitay/hbase-git/bin/../logs, -Dhbase.log.file=hbase-nitay-regionserver-aa0-000-12.u.powerset.com.log, -Dhbase.home.dir=/home/nitay/hbase-git/bin/.., -Dhbase.id.str=nitay, -Dhbase.root.logger=INFO,DRFA, -Djava.library.path=/home/nitay/hbase-git/bin/../lib/native/Linux-amd64-64] 2009-03-30 07:55:18,150 INFO org.apache.hadoop.hbase.regionserver.MemcacheFlusher: globalMemcacheLimit=355.6m, globalMemcacheLimitLowMark=222.2m, maxHeap=888.9m 2009-03-30 07:55:18,155 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Runs every 10000000ms 2009-03-30 07:55:18,195 INFO org.apache.hadoop.hbase.ipc.HBaseRpcMetrics: Initializing RPC Metrics with hostName=HRegionServer, port=60027 2009-03-30 07:55:18,274 INFO org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Quorum servers: aa0-000-14.u.powerset.com:2188,aa0-000-13.u.powerset.com:2188,aa0-000-12.u.powerset.com:2188,aa0-001-13.u.powerset.com:2188,aa0-000-15.u.powerset.com:2188 2009-03-30 07:55:18,433 WARN org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Failed to set watcher on ZNode /hbase/master org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss at org.apache.zookeeper.KeeperException.create(KeeperException.java:90) at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:709) at org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.watchMasterAddress(ZooKeeperWrapper.java:235) at org.apache.hadoop.hbase.regionserver.HRegionServer.watchMasterAddress(HRegionServer.java:344) at org.apache.hadoop.hbase.regionserver.HRegionServer.(HRegionServer.java:299) at org.apache.hadoop.hbase.regionserver.HRegionServer.(HRegionServer.java:232) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown Source) at java.lang.reflect.Constructor.newInstance(Unknown Source) at org.apache.hadoop.hbase.regionserver.HRegionServer.doMain(HRegionServer.java:2313) at org.apache.hadoop.hbase.regionserver.HRegionServer.main(HRegionServer.java:2345) 2009-03-30 07:55:18,435 WARN org.apache.hadoop.hbase.regionserver.HRegionServer: Unable to set watcher on ZooKeeper master address. Retrying. 2009-03-30 07:55:21,450 DEBUG org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Set watcher on master address ZNode /hbase/master 2009-03-30 07:55:21,471 DEBUG org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Created ZNode /hbase/rs/1238399721450 with data 0.0.0.0 2009-03-30 07:55:21,508 DEBUG org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Read ZNode /hbase/master got 208.76.44.158:60007 2009-03-30 07:55:21,509 DEBUG org.apache.hadoop.hbase.regionserver.HRegionServer: Telling master at 208.76.44.158:60007 that we are up 2009-03-30 07:55:28,035 DEBUG org.apache.hadoop.hbase.regionserver.HRegionServer: Got ZooKeeper event, state: SyncConnected, type: NodeDeleted, path: /hbase/master 2009-03-30 07:55:28,036 DEBUG org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Set watcher on master address ZNode /hbase/master 2009-03-30 07:55:28,057 DEBUG org.apache.hadoop.hbase.regionserver.HRegionServer: Got ZooKeeper event, state: SyncConnected, type: NodeCreated, path: /hbase/master 2009-03-30 07:55:28,058 DEBUG org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Read ZNode /hbase/master got 208.76.44.158:60007 2009-03-30 07:55:28,058 DEBUG org.apache.hadoop.hbase.regionserver.HRegionServer: Telling master at 208.76.44.158:60007 that we are up 2009-03-30 07:55:28,915 DEBUG org.apache.hadoop.hbase.regionserver.HRegionServer: sending initial server load: requests=0, regions=0, usedHeap=21, maxHeap=888 2009-03-30 07:55:28,924 DEBUG org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Set watcher on master address ZNode /hbase/master 2009-03-30 07:55:28,966 DEBUG org.apache.hadoop.hbase.regionserver.HRegionServer: Config from master: hbase.regionserver.address=208.76.44.139 2009-03-30 07:55:28,966 DEBUG org.apache.hadoop.hbase.regionserver.HRegionServer: Config from master: fs.default.name=hdfs://aa0-001-13.u.powerset.com:9007/hbase 2009-03-30 07:55:28,966 DEBUG org.apache.hadoop.hbase.regionserver.HRegionServer: Config from master: hbase.rootdir=hdfs://aa0-001-13.u.powerset.com:9007/hbase 2009-03-30 07:55:29,158 DEBUG org.apache.hadoop.hbase.regionserver.HRegionServer: Log dir hdfs://aa0-001-13.u.powerset.com:9007/hbase/log_aa0-000-12.u.powerset.com_1238399721450_60027 2009-03-30 07:55:29,233 INFO org.apache.hadoop.hbase.regionserver.HLog: New log writer: /hbase/log_aa0-000-12.u.powerset.com_1238399721450_60027/hlog.dat.1238399729174 2009-03-30 07:55:29,237 INFO org.apache.hadoop.metrics.jvm.JvmMetrics: Initializing JVM Metrics with processName=RegionServer, sessionId=regionserver/0:0:0:0:0:0:0:0:60027 2009-03-30 07:55:29,238 INFO org.apache.hadoop.hbase.regionserver.metrics.RegionServerMetrics: Initialized 2009-03-30 07:55:29,372 INFO org.mortbay.util.Credential: Checking Resource aliases 2009-03-30 07:55:29,377 INFO org.mortbay.http.HttpServer: Version Jetty/5.1.4 2009-03-30 07:55:29,378 INFO org.mortbay.util.Container: Started HttpContext[/logs,/logs] 2009-03-30 07:55:29,704 INFO org.mortbay.util.Container: Started org.mortbay.jetty.servlet.WebApplicationHandler@309ff0a8 2009-03-30 07:55:29,747 INFO org.mortbay.util.Container: Started WebApplicationContext[/static,/static] 2009-03-30 07:55:29,973 INFO org.mortbay.util.Container: Started org.mortbay.jetty.servlet.WebApplicationHandler@342798e7 2009-03-30 07:55:29,977 INFO org.mortbay.util.Container: Started WebApplicationContext[/,/] 2009-03-30 07:55:30,006 INFO org.mortbay.http.SocketListener: Started SocketListener on 0.0.0.0:60037 2009-03-30 07:55:30,006 INFO org.mortbay.util.Container: Started org.mortbay.jetty.Server@49bdc9d8 2009-03-30 07:55:30,006 DEBUG org.apache.hadoop.hbase.regionserver.HRegionServer: Waiting to exit safe mode 2009-03-30 07:55:30,007 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server Responder: starting 2009-03-30 07:55:30,007 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server listener on 60027: starting 2009-03-30 07:55:30,008 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 0 on 60027: starting 2009-03-30 07:55:30,009 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 1 on 60027: starting 2009-03-30 07:55:30,009 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 2 on 60027: starting 2009-03-30 07:55:30,009 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 3 on 60027: starting 2009-03-30 07:55:30,009 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 4 on 60027: starting 2009-03-30 07:55:30,009 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 5 on 60027: starting 2009-03-30 07:55:30,010 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 6 on 60027: starting 2009-03-30 07:55:30,010 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 7 on 60027: starting 2009-03-30 07:55:30,010 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 8 on 60027: starting 2009-03-30 07:55:30,010 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 9 on 60027: starting 2009-03-30 07:55:30,010 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: HRegionServer started at: 208.76.44.139:60027 2009-03-30 07:55:30,011 DEBUG org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Read ZNode /hbase/root-region-server got 208.76.44.141:60027 2009-03-30 07:55:36,041 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: MSG_REGION_OPEN: .META.,,1 2009-03-30 07:55:36,045 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Worker: MSG_REGION_OPEN: .META.,,1 2009-03-30 07:55:36,056 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Opening region .META.,,1/1028785192 2009-03-30 07:55:36,078 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Next sequence id for region .META.,,1 is 0 2009-03-30 07:55:36,081 INFO org.apache.hadoop.hbase.regionserver.HRegion: region .META.,,1/1028785192 available 2009-03-30 07:55:36,081 DEBUG org.apache.hadoop.hbase.regionserver.CompactSplitThread: Compaction requested for region .META.,,1/1028785192 because: Region open check 2009-03-30 07:55:49,247 INFO org.apache.hadoop.hbase.regionserver.HRegion: starting compaction on region .META.,,1 2009-03-30 07:55:49,268 DEBUG org.apache.hadoop.hbase.regionserver.Store: 1028785192/historian: no store files to compact 2009-03-30 07:55:49,268 DEBUG org.apache.hadoop.hbase.regionserver.Store: 1028785192/info: no store files to compact 2009-03-30 07:55:49,269 INFO org.apache.hadoop.hbase.regionserver.HRegion: compaction completed on region .META.,,1 in 0sec 2009-03-30 07:56:12,198 WARN org.apache.hadoop.hbase.regionserver.HRegionServer: Processing message (Retry: 0) java.io.IOException: Call to /208.76.44.158:60007 failed on local exception: java.io.EOFException at org.apache.hadoop.hbase.ipc.HBaseClient.wrapException(HBaseClient.java:752) at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:722) at org.apache.hadoop.hbase.ipc.HBaseRPC$Invoker.invoke(HBaseRPC.java:322) at $Proxy0.regionServerReport(Unknown Source) at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:398) at java.lang.Thread.run(Unknown Source) Caused by: java.io.EOFException at java.io.DataInputStream.readInt(Unknown Source) at org.apache.hadoop.hbase.ipc.HBaseClient$Connection.receiveResponse(HBaseClient.java:501) at org.apache.hadoop.hbase.ipc.HBaseClient$Connection.run(HBaseClient.java:445) 2009-03-30 07:56:13,207 INFO org.apache.hadoop.ipc.HBaseClass: Retrying connect to server: /208.76.44.158:60007. Already tried 0 time(s). 2009-03-30 07:56:14,217 INFO org.apache.hadoop.ipc.HBaseClass: Retrying connect to server: /208.76.44.158:60007. Already tried 1 time(s). 2009-03-30 07:56:15,237 INFO org.apache.hadoop.ipc.HBaseClass: Retrying connect to server: /208.76.44.158:60007. Already tried 2 time(s). 2009-03-30 07:56:16,257 INFO org.apache.hadoop.ipc.HBaseClass: Retrying connect to server: /208.76.44.158:60007. Already tried 3 time(s). 2009-03-30 07:56:17,277 INFO org.apache.hadoop.ipc.HBaseClass: Retrying connect to server: /208.76.44.158:60007. Already tried 4 time(s). 2009-03-30 07:56:18,297 INFO org.apache.hadoop.ipc.HBaseClass: Retrying connect to server: /208.76.44.158:60007. Already tried 5 time(s). 2009-03-30 07:56:19,307 INFO org.apache.hadoop.ipc.HBaseClass: Retrying connect to server: /208.76.44.158:60007. Already tried 6 time(s). 2009-03-30 07:56:20,317 INFO org.apache.hadoop.ipc.HBaseClass: Retrying connect to server: /208.76.44.158:60007. Already tried 7 time(s). 2009-03-30 07:56:21,327 INFO org.apache.hadoop.ipc.HBaseClass: Retrying connect to server: /208.76.44.158:60007. Already tried 8 time(s). 2009-03-30 07:56:22,042 DEBUG org.apache.hadoop.hbase.regionserver.HRegionServer: Got ZooKeeper event, state: SyncConnected, type: NodeDeleted, path: /hbase/master 2009-03-30 07:56:22,043 DEBUG org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Set watcher on master address ZNode /hbase/master 2009-03-30 07:56:22,337 INFO org.apache.hadoop.ipc.HBaseClass: Retrying connect to server: /208.76.44.158:60007. Already tried 9 time(s). 2009-03-30 07:56:22,338 WARN org.apache.hadoop.hbase.regionserver.HRegionServer: Processing message (Retry: 1) java.net.ConnectException: Call to /208.76.44.158:60007 failed on connection exception: java.net.ConnectException: Connection refused at org.apache.hadoop.hbase.ipc.HBaseClient.wrapException(HBaseClient.java:744) at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:722) at org.apache.hadoop.hbase.ipc.HBaseRPC$Invoker.invoke(HBaseRPC.java:322) at $Proxy0.regionServerReport(Unknown Source) at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:398) at java.lang.Thread.run(Unknown Source) Caused by: java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(Unknown Source) at sun.nio.ch.SocketAdaptor.connect(Unknown Source) at org.apache.hadoop.hbase.ipc.HBaseClient$Connection.setupIOstreams(HBaseClient.java:305) at org.apache.hadoop.hbase.ipc.HBaseClient.getConnection(HBaseClient.java:826) at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:707) ... 4 more 2009-03-30 07:56:23,347 INFO org.apache.hadoop.ipc.HBaseClass: Retrying connect to server: /208.76.44.158:60007. Already tried 0 time(s). 2009-03-30 07:56:24,357 INFO org.apache.hadoop.ipc.HBaseClass: Retrying connect to server: /208.76.44.158:60007. Already tried 1 time(s). 2009-03-30 07:56:25,367 INFO org.apache.hadoop.ipc.HBaseClass: Retrying connect to server: /208.76.44.158:60007. Already tried 2 time(s). 2009-03-30 07:56:26,377 INFO org.apache.hadoop.ipc.HBaseClass: Retrying connect to server: /208.76.44.158:60007. Already tried 3 time(s). 2009-03-30 07:56:27,387 INFO org.apache.hadoop.ipc.HBaseClass: Retrying connect to server: /208.76.44.158:60007. Already tried 4 time(s). 2009-03-30 07:56:28,397 INFO org.apache.hadoop.ipc.HBaseClass: Retrying connect to server: /208.76.44.158:60007. Already tried 5 time(s). 2009-03-30 07:56:29,407 INFO org.apache.hadoop.ipc.HBaseClass: Retrying connect to server: /208.76.44.158:60007. Already tried 6 time(s). 2009-03-30 07:56:30,417 INFO org.apache.hadoop.ipc.HBaseClass: Retrying connect to server: /208.76.44.158:60007. Already tried 7 time(s). 2009-03-30 07:56:31,427 INFO org.apache.hadoop.ipc.HBaseClass: Retrying connect to server: /208.76.44.158:60007. Already tried 8 time(s). 2009-03-30 07:56:32,447 INFO org.apache.hadoop.ipc.HBaseClass: Retrying connect to server: /208.76.44.158:60007. Already tried 9 time(s). 2009-03-30 07:56:32,448 WARN org.apache.hadoop.hbase.regionserver.HRegionServer: Processing message (Retry: 2) java.net.ConnectException: Call to /208.76.44.158:60007 failed on connection exception: java.net.ConnectException: Connection refused at org.apache.hadoop.hbase.ipc.HBaseClient.wrapException(HBaseClient.java:744) at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:722) at org.apache.hadoop.hbase.ipc.HBaseRPC$Invoker.invoke(HBaseRPC.java:322) at $Proxy0.regionServerReport(Unknown Source) at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:398) at java.lang.Thread.run(Unknown Source) Caused by: java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(Unknown Source) at sun.nio.ch.SocketAdaptor.connect(Unknown Source) at org.apache.hadoop.hbase.ipc.HBaseClient$Connection.setupIOstreams(HBaseClient.java:305) at org.apache.hadoop.hbase.ipc.HBaseClient.getConnection(HBaseClient.java:826) at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:707) ... 4 more 2009-03-30 07:56:33,457 INFO org.apache.hadoop.ipc.HBaseClass: Retrying connect to server: /208.76.44.158:60007. Already tried 0 time(s). 2009-03-30 07:56:34,477 INFO org.apache.hadoop.ipc.HBaseClass: Retrying connect to server: /208.76.44.158:60007. Already tried 1 time(s). 2009-03-30 07:56:35,497 INFO org.apache.hadoop.ipc.HBaseClass: Retrying connect to server: /208.76.44.158:60007. Already tried 2 time(s). 2009-03-30 07:56:36,517 INFO org.apache.hadoop.ipc.HBaseClass: Retrying connect to server: /208.76.44.158:60007. Already tried 3 time(s). 2009-03-30 07:56:37,537 INFO org.apache.hadoop.ipc.HBaseClass: Retrying connect to server: /208.76.44.158:60007. Already tried 4 time(s). 2009-03-30 07:56:38,557 INFO org.apache.hadoop.ipc.HBaseClass: Retrying connect to server: /208.76.44.158:60007. Already tried 5 time(s). 2009-03-30 07:56:39,567 INFO org.apache.hadoop.ipc.HBaseClass: Retrying connect to server: /208.76.44.158:60007. Already tried 6 time(s). 2009-03-30 07:56:40,577 INFO org.apache.hadoop.ipc.HBaseClass: Retrying connect to server: /208.76.44.158:60007. Already tried 7 time(s). 2009-03-30 07:56:41,587 INFO org.apache.hadoop.ipc.HBaseClass: Retrying connect to server: /208.76.44.158:60007. Already tried 8 time(s). 2009-03-30 07:56:42,607 INFO org.apache.hadoop.ipc.HBaseClass: Retrying connect to server: /208.76.44.158:60007. Already tried 9 time(s). 2009-03-30 07:56:42,608 WARN org.apache.hadoop.hbase.regionserver.HRegionServer: Processing message (Retry: 3) java.net.ConnectException: Call to /208.76.44.158:60007 failed on connection exception: java.net.ConnectException: Connection refused at org.apache.hadoop.hbase.ipc.HBaseClient.wrapException(HBaseClient.java:744) at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:722) at org.apache.hadoop.hbase.ipc.HBaseRPC$Invoker.invoke(HBaseRPC.java:322) at $Proxy0.regionServerReport(Unknown Source) at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:398) at java.lang.Thread.run(Unknown Source) Caused by: java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(Unknown Source) at sun.nio.ch.SocketAdaptor.connect(Unknown Source) at org.apache.hadoop.hbase.ipc.HBaseClient$Connection.setupIOstreams(HBaseClient.java:305) at org.apache.hadoop.hbase.ipc.HBaseClient.getConnection(HBaseClient.java:826) at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:707) ... 4 more 2009-03-30 07:56:43,627 INFO org.apache.hadoop.ipc.HBaseClass: Retrying connect to server: /208.76.44.158:60007. Already tried 0 time(s). 2009-03-30 07:56:44,647 INFO org.apache.hadoop.ipc.HBaseClass: Retrying connect to server: /208.76.44.158:60007. Already tried 1 time(s). 2009-03-30 07:56:45,657 INFO org.apache.hadoop.ipc.HBaseClass: Retrying connect to server: /208.76.44.158:60007. Already tried 2 time(s). 2009-03-30 07:56:46,667 INFO org.apache.hadoop.ipc.HBaseClass: Retrying connect to server: /208.76.44.158:60007. Already tried 3 time(s). 2009-03-30 07:56:47,677 INFO org.apache.hadoop.ipc.HBaseClass: Retrying connect to server: /208.76.44.158:60007. Already tried 4 time(s). 2009-03-30 07:56:48,687 INFO org.apache.hadoop.ipc.HBaseClass: Retrying connect to server: /208.76.44.158:60007. Already tried 5 time(s). 2009-03-30 07:56:49,707 INFO org.apache.hadoop.ipc.HBaseClass: Retrying connect to server: /208.76.44.158:60007. Already tried 6 time(s). 2009-03-30 07:56:50,717 INFO org.apache.hadoop.ipc.HBaseClass: Retrying connect to server: /208.76.44.158:60007. Already tried 7 time(s). 2009-03-30 07:56:51,727 INFO org.apache.hadoop.ipc.HBaseClass: Retrying connect to server: /208.76.44.158:60007. Already tried 8 time(s). 2009-03-30 07:56:52,737 INFO org.apache.hadoop.ipc.HBaseClass: Retrying connect to server: /208.76.44.158:60007. Already tried 9 time(s). 2009-03-30 07:56:52,738 WARN org.apache.hadoop.hbase.regionserver.HRegionServer: Processing message (Retry: 4) java.net.ConnectException: Call to /208.76.44.158:60007 failed on connection exception: java.net.ConnectException: Connection refused at org.apache.hadoop.hbase.ipc.HBaseClient.wrapException(HBaseClient.java:744) at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:722) at org.apache.hadoop.hbase.ipc.HBaseRPC$Invoker.invoke(HBaseRPC.java:322) at $Proxy0.regionServerReport(Unknown Source) at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:398) at java.lang.Thread.run(Unknown Source) Caused by: java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(Unknown Source) at sun.nio.ch.SocketAdaptor.connect(Unknown Source) at org.apache.hadoop.hbase.ipc.HBaseClient$Connection.setupIOstreams(HBaseClient.java:305) at org.apache.hadoop.hbase.ipc.HBaseClient.getConnection(HBaseClient.java:826) at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:707) ... 4 more 2009-03-30 07:56:53,747 INFO org.apache.hadoop.ipc.HBaseClass: Retrying connect to server: /208.76.44.158:60007. Already tried 0 time(s). 2009-03-30 07:56:54,757 INFO org.apache.hadoop.ipc.HBaseClass: Retrying connect to server: /208.76.44.158:60007. Already tried 1 time(s). 2009-03-30 07:56:55,767 INFO org.apache.hadoop.ipc.HBaseClass: Retrying connect to server: /208.76.44.158:60007. Already tried 2 time(s). 2009-03-30 07:56:56,087 DEBUG org.apache.hadoop.hbase.regionserver.HRegionServer: setting compaction limit to 2 2009-03-30 07:56:56,777 INFO org.apache.hadoop.ipc.HBaseClass: Retrying connect to server: /208.76.44.158:60007. Already tried 3 time(s). 2009-03-30 07:56:57,787 INFO org.apache.hadoop.ipc.HBaseClass: Retrying connect to server: /208.76.44.158:60007. Already tried 4 time(s). 2009-03-30 07:56:58,797 INFO org.apache.hadoop.ipc.HBaseClass: Retrying connect to server: /208.76.44.158:60007. Already tried 5 time(s). 2009-03-30 07:56:59,807 INFO org.apache.hadoop.ipc.HBaseClass: Retrying connect to server: /208.76.44.158:60007. Already tried 6 time(s). 2009-03-30 07:57:00,827 INFO org.apache.hadoop.ipc.HBaseClass: Retrying connect to server: /208.76.44.158:60007. Already tried 7 time(s). 2009-03-30 07:57:01,837 INFO org.apache.hadoop.ipc.HBaseClass: Retrying connect to server: /208.76.44.158:60007. Already tried 8 time(s). 2009-03-30 07:57:02,847 INFO org.apache.hadoop.ipc.HBaseClass: Retrying connect to server: /208.76.44.158:60007. Already tried 9 time(s). 2009-03-30 07:57:02,848 WARN org.apache.hadoop.hbase.regionserver.HRegionServer: Processing message (Retry: 5) java.net.ConnectException: Call to /208.76.44.158:60007 failed on connection exception: java.net.ConnectException: Connection refused at org.apache.hadoop.hbase.ipc.HBaseClient.wrapException(HBaseClient.java:744) at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:722) at org.apache.hadoop.hbase.ipc.HBaseRPC$Invoker.invoke(HBaseRPC.java:322) at $Proxy0.regionServerReport(Unknown Source) at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:398) at java.lang.Thread.run(Unknown Source) Caused by: java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(Unknown Source) at sun.nio.ch.SocketAdaptor.connect(Unknown Source) at org.apache.hadoop.hbase.ipc.HBaseClient$Connection.setupIOstreams(HBaseClient.java:305) at org.apache.hadoop.hbase.ipc.HBaseClient.getConnection(HBaseClient.java:826) at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:707) ... 4 more 2009-03-30 07:57:03,857 INFO org.apache.hadoop.ipc.HBaseClass: Retrying connect to server: /208.76.44.158:60007. Already tried 0 time(s). 2009-03-30 07:57:04,867 INFO org.apache.hadoop.ipc.HBaseClass: Retrying connect to server: /208.76.44.158:60007. Already tried 1 time(s). 2009-03-30 07:57:05,878 INFO org.apache.hadoop.ipc.HBaseClass: Retrying connect to server: /208.76.44.158:60007. Already tried 2 time(s). 2009-03-30 07:57:06,897 INFO org.apache.hadoop.ipc.HBaseClass: Retrying connect to server: /208.76.44.158:60007. Already tried 3 time(s). 2009-03-30 07:57:07,907 INFO org.apache.hadoop.ipc.HBaseClass: Retrying connect to server: /208.76.44.158:60007. Already tried 4 time(s). 2009-03-30 07:57:08,917 INFO org.apache.hadoop.ipc.HBaseClass: Retrying connect to server: /208.76.44.158:60007. Already tried 5 time(s). 2009-03-30 07:57:09,927 INFO org.apache.hadoop.ipc.HBaseClass: Retrying connect to server: /208.76.44.158:60007. Already tried 6 time(s). 2009-03-30 07:57:10,947 INFO org.apache.hadoop.ipc.HBaseClass: Retrying connect to server: /208.76.44.158:60007. Already tried 7 time(s). 2009-03-30 07:57:11,957 INFO org.apache.hadoop.ipc.HBaseClass: Retrying connect to server: /208.76.44.158:60007. Already tried 8 time(s). 2009-03-30 07:57:12,967 INFO org.apache.hadoop.ipc.HBaseClass: Retrying connect to server: /208.76.44.158:60007. Already tried 9 time(s). 2009-03-30 07:57:12,968 WARN org.apache.hadoop.hbase.regionserver.HRegionServer: Processing message (Retry: 6) java.net.ConnectException: Call to /208.76.44.158:60007 failed on connection exception: java.net.ConnectException: Connection refused at org.apache.hadoop.hbase.ipc.HBaseClient.wrapException(HBaseClient.java:744) at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:722) at org.apache.hadoop.hbase.ipc.HBaseRPC$Invoker.invoke(HBaseRPC.java:322) at $Proxy0.regionServerReport(Unknown Source) at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:398) at java.lang.Thread.run(Unknown Source) Caused by: java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(Unknown Source) at sun.nio.ch.SocketAdaptor.connect(Unknown Source) at org.apache.hadoop.hbase.ipc.HBaseClient$Connection.setupIOstreams(HBaseClient.java:305) at org.apache.hadoop.hbase.ipc.HBaseClient.getConnection(HBaseClient.java:826) at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:707) ... 4 more 2009-03-30 07:57:13,977 INFO org.apache.hadoop.ipc.HBaseClass: Retrying connect to server: /208.76.44.158:60007. Already tried 0 time(s). 2009-03-30 07:57:14,987 INFO org.apache.hadoop.ipc.HBaseClass: Retrying connect to server: /208.76.44.158:60007. Already tried 1 time(s). 2009-03-30 07:57:15,232 DEBUG org.apache.hadoop.hbase.regionserver.HRegionServer: Got ZooKeeper event, state: SyncConnected, type: NodeCreated, path: /hbase/master 2009-03-30 07:57:15,233 DEBUG org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Read ZNode /hbase/master got 208.76.44.139:60007 2009-03-30 07:57:15,233 DEBUG org.apache.hadoop.hbase.regionserver.HRegionServer: Telling master at 208.76.44.139:60007 that we are up 2009-03-30 07:57:15,997 INFO org.apache.hadoop.ipc.HBaseClass: Retrying connect to server: /208.76.44.158:60007. Already tried 2 time(s). 2009-03-30 07:57:16,123 DEBUG org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Set watcher on master address ZNode /hbase/master 2009-03-30 07:57:17,007 INFO org.apache.hadoop.ipc.HBaseClass: Retrying connect to server: /208.76.44.158:60007. Already tried 3 time(s). 2009-03-30 07:57:18,017 INFO org.apache.hadoop.ipc.HBaseClass: Retrying connect to server: /208.76.44.158:60007. Already tried 4 time(s). 2009-03-30 07:57:19,027 INFO org.apache.hadoop.ipc.HBaseClass: Retrying connect to server: /208.76.44.158:60007. Already tried 5 time(s). 2009-03-30 07:57:20,037 INFO org.apache.hadoop.ipc.HBaseClass: Retrying connect to server: /208.76.44.158:60007. Already tried 6 time(s). 2009-03-30 07:57:21,047 INFO org.apache.hadoop.ipc.HBaseClass: Retrying connect to server: /208.76.44.158:60007. Already tried 7 time(s). 2009-03-30 07:57:22,057 INFO org.apache.hadoop.ipc.HBaseClass: Retrying connect to server: /208.76.44.158:60007. Already tried 8 time(s). 2009-03-30 07:57:23,077 INFO org.apache.hadoop.ipc.HBaseClass: Retrying connect to server: /208.76.44.158:60007. Already tried 9 time(s). 2009-03-30 07:57:23,078 WARN org.apache.hadoop.hbase.regionserver.HRegionServer: Processing message (Retry: 7) java.net.ConnectException: Call to /208.76.44.158:60007 failed on connection exception: java.net.ConnectException: Connection refused at org.apache.hadoop.hbase.ipc.HBaseClient.wrapException(HBaseClient.java:744) at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:722) at org.apache.hadoop.hbase.ipc.HBaseRPC$Invoker.invoke(HBaseRPC.java:322) at $Proxy0.regionServerReport(Unknown Source) at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:398) at java.lang.Thread.run(Unknown Source) Caused by: java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(Unknown Source) at sun.nio.ch.SocketAdaptor.connect(Unknown Source) at org.apache.hadoop.hbase.ipc.HBaseClient$Connection.setupIOstreams(HBaseClient.java:305) at org.apache.hadoop.hbase.ipc.HBaseClient.getConnection(HBaseClient.java:826) at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:707) ... 4 more 2009-03-30 07:57:23,083 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: MSG_CALL_SERVER_STARTUP 2009-03-30 07:57:23,086 DEBUG org.apache.hadoop.hbase.regionserver.HRegionServer: closing region .META.,,1 2009-03-30 07:57:23,086 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Closing .META.,,1: compactions & flushes disabled 2009-03-30 07:57:23,086 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Updates disabled for region, no outstanding scanners on .META.,,1 2009-03-30 07:57:23,086 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: No more row locks outstanding on region .META.,,1 2009-03-30 07:57:23,086 DEBUG org.apache.hadoop.hbase.regionserver.Store: closed 1028785192/historian 2009-03-30 07:57:23,086 DEBUG org.apache.hadoop.hbase.regionserver.Store: closed 1028785192/info 2009-03-30 07:57:23,086 INFO org.apache.hadoop.hbase.regionserver.HRegion: Closed .META.,,1 2009-03-30 07:57:23,086 DEBUG org.apache.hadoop.hbase.regionserver.HLog: closing log writer in hdfs://aa0-001-13.u.powerset.com:9007/hbase/log_aa0-000-12.u.powerset.com_1238399721450_60027 2009-03-30 07:57:23,159 INFO org.apache.hadoop.hdfs.DFSClient: Exception in createBlockOutputStream java.io.IOException: Could not read from stream 2009-03-30 07:57:23,159 INFO org.apache.hadoop.hdfs.DFSClient: Abandoning block blk_5913848576688223858_1322 2009-03-30 07:57:29,170 INFO org.apache.hadoop.hdfs.DFSClient: Exception in createBlockOutputStream java.io.IOException: Could not read from stream 2009-03-30 07:57:29,171 INFO org.apache.hadoop.hdfs.DFSClient: Abandoning block blk_5164772576096516141_1322 2009-03-30 07:57:35,181 INFO org.apache.hadoop.hdfs.DFSClient: Exception in createBlockOutputStream java.io.IOException: Could not read from stream 2009-03-30 07:57:35,181 INFO org.apache.hadoop.hdfs.DFSClient: Abandoning block blk_-4466434869304452067_1322 2009-03-30 07:57:41,191 INFO org.apache.hadoop.hdfs.DFSClient: Exception in createBlockOutputStream java.io.IOException: Could not read from stream 2009-03-30 07:57:41,191 INFO org.apache.hadoop.hdfs.DFSClient: Abandoning block blk_1664120410999456709_1322 2009-03-30 07:57:47,197 WARN org.apache.hadoop.hdfs.DFSClient: DataStreamer Exception: java.io.IOException: Unable to create new block. at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2723) at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2000(DFSClient.java:1997) at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2183) 2009-03-30 07:57:47,197 WARN org.apache.hadoop.hdfs.DFSClient: Error Recovery for block blk_1664120410999456709_1322 bad datanode[0] nodes == null 2009-03-30 07:57:47,198 WARN org.apache.hadoop.hdfs.DFSClient: Could not get block locations. Aborting... 2009-03-30 07:57:47,198 ERROR org.apache.hadoop.hbase.regionserver.HRegionServer: error closing and deleting HLog java.io.IOException: Could not read from stream at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:119) at java.io.DataInputStream.readByte(Unknown Source) at org.apache.hadoop.io.WritableUtils.readVLong(WritableUtils.java:325) at org.apache.hadoop.io.WritableUtils.readVInt(WritableUtils.java:346) at org.apache.hadoop.io.Text.readString(Text.java:400) at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.createBlockOutputStream(DFSClient.java:2779) at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2704) at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2000(DFSClient.java:1997) at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2183) 2009-03-30 07:57:47,200 DEBUG org.apache.hadoop.hbase.regionserver.HRegionServer: Log dir hdfs://aa0-001-13.u.powerset.com:9007/hbase/log_aa0-000-12.u.powerset.com_1238399867198_60027 2009-03-30 07:57:47,213 INFO org.apache.hadoop.hbase.regionserver.HLog: New log writer: /hbase/log_aa0-000-12.u.powerset.com_1238399867198_60027/hlog.dat.1238399867209 2009-03-30 07:57:47,214 DEBUG org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Read ZNode /hbase/master got 208.76.44.139:60007 2009-03-30 07:57:47,214 DEBUG org.apache.hadoop.hbase.regionserver.HRegionServer: Telling master at 208.76.44.139:60007 that we are up 2009-03-30 07:57:47,218 DEBUG org.apache.hadoop.hbase.regionserver.HRegionServer: sending initial server load: requests=0, regions=0, usedHeap=33, maxHeap=888 2009-03-30 07:57:47,267 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: MSG_REGION_OPEN: -ROOT-,,0 2009-03-30 07:57:47,268 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Worker: MSG_REGION_OPEN: -ROOT-,,0 2009-03-30 07:57:47,269 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Opening region -ROOT-,,0/70236052 2009-03-30 07:57:47,277 WARN org.apache.hadoop.hbase.regionserver.Store: Skipping hdfs://aa0-001-13.u.powerset.com:9007/hbase/-ROOT-/70236052/info/1679267180657957681 because its empty. HBASE-646 DATA LOSS? 2009-03-30 07:57:47,278 WARN org.apache.hadoop.hbase.regionserver.Store: Skipping hdfs://aa0-001-13.u.powerset.com:9007/hbase/-ROOT-/70236052/info/3917849361756330712 because its empty. HBASE-646 DATA LOSS? 2009-03-30 07:57:47,279 WARN org.apache.hadoop.hbase.regionserver.Store: Skipping hdfs://aa0-001-13.u.powerset.com:9007/hbase/-ROOT-/70236052/info/4053850038924430270 because its empty. HBASE-646 DATA LOSS? 2009-03-30 07:57:47,281 WARN org.apache.hadoop.hbase.regionserver.Store: Skipping hdfs://aa0-001-13.u.powerset.com:9007/hbase/-ROOT-/70236052/info/6503728984787304918 because its empty. HBASE-646 DATA LOSS? 2009-03-30 07:57:47,302 DEBUG org.apache.hadoop.hbase.regionserver.Store: loaded /hbase/-ROOT-/70236052/info/7991179185768818530, isReference=false, sequence id=1, length=856, majorCompaction=true 2009-03-30 07:57:47,303 DEBUG org.apache.hadoop.hbase.regionserver.Store: Loaded 1 file(s) in Store 70236052/info, max sequence id 1 2009-03-30 07:57:47,304 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Next sequence id for region -ROOT-,,0 is 2 2009-03-30 07:57:47,306 INFO org.apache.hadoop.hbase.regionserver.HRegion: region -ROOT-,,0/70236052 available 2009-03-30 07:57:47,306 DEBUG org.apache.hadoop.hbase.regionserver.CompactSplitThread: Compaction requested for region -ROOT-,,0/70236052 because: Region open check 2009-03-30 07:57:47,306 DEBUG org.apache.hadoop.hbase.regionserver.HLog: changing sequence number from 0 to 2 2009-03-30 07:57:47,307 INFO org.apache.hadoop.hbase.regionserver.HRegion: starting compaction on region -ROOT-,,0 2009-03-30 07:57:47,312 DEBUG org.apache.hadoop.hbase.regionserver.Store: Skipping major compaction of 70236052/info because one (major) compacted file only and elapsedTime 386918011ms is < ttl=-1 2009-03-30 07:57:47,313 INFO org.apache.hadoop.hbase.regionserver.HRegion: compaction completed on region -ROOT-,,0 in 0sec 2009-03-30 07:58:56,157 DEBUG org.apache.hadoop.hbase.regionserver.HRegionServer: setting compaction limit to 3 2009-03-30 08:01:36,257 DEBUG org.apache.hadoop.hbase.regionserver.HRegionServer: setting compaction limit to 4 2009-03-30 08:04:56,367 DEBUG org.apache.hadoop.hbase.regionserver.HRegionServer: setting compaction limit to -1 2009-03-30 08:05:16,377 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: compactions no longer limited 2009-03-30 10:42:09,270 DEBUG org.apache.hadoop.hbase.regionserver.Store: Skipping major compaction of 70236052/info because one (major) compacted file only and elapsedTime 396779969ms is < ttl=-1 2009-03-30 13:28:49,270 DEBUG org.apache.hadoop.hbase.regionserver.Store: Skipping major compaction of 70236052/info because one (major) compacted file only and elapsedTime 406779969ms is < ttl=-1 2009-03-30 16:15:29,271 DEBUG org.apache.hadoop.hbase.regionserver.Store: Skipping major compaction of 70236052/info because one (major) compacted file only and elapsedTime 416779969ms is < ttl=-1 2009-03-30 19:02:09,271 DEBUG org.apache.hadoop.hbase.regionserver.Store: Skipping major compaction of 70236052/info because one (major) compacted file only and elapsedTime 426779970ms is < ttl=-1