2010-10-27 20:40:50,622 INFO [RS_CLOSE_REGION-192.168.0.44,55804,1288237198579-0] regionserver.HRegion(531): Closed tableRestart,s\xD0jR1,1288237107591.bfecfe09e368b7b20d0291f88ac43620. 2010-10-27 20:40:50,622 DEBUG [RS_CLOSE_REGION-192.168.0.44,55804,1288237198579-0] handler.CloseRegionHandler(128): Closed region tableRestart,s\xD0jR1,1288237107591.bfecfe09e368b7b20d0291f88ac43620. 2010-10-27 20:40:50,622 DEBUG [RS_CLOSE_REGION-192.168.0.44,55804,1288237198579-0] handler.CloseRegionHandler(99): Processing close of tableRestart,a\x8Er:\xD1,1288237103699.e8c41668483c8d9ae10681b14be0b2c3. 2010-10-27 20:40:50,622 DEBUG [RS_CLOSE_REGION-192.168.0.44,55804,1288237198579-0] regionserver.HRegion(492): Closing tableRestart,a\x8Er:\xD1,1288237103699.e8c41668483c8d9ae10681b14be0b2c3.: disabling compactions & flushes 2010-10-27 20:40:50,623 DEBUG [RS_CLOSE_REGION-192.168.0.44,55804,1288237198579-0] regionserver.HRegion(520): Updates disabled for region tableRestart,a\x8Er:\xD1,1288237103699.e8c41668483c8d9ae10681b14be0b2c3. 2010-10-27 20:40:50,623 DEBUG [RS_CLOSE_REGION-192.168.0.44,55804,1288237198579-0] regionserver.Store(412): closed family 2010-10-27 20:40:50,567 DEBUG [RS_CLOSE_REGION-192.168.0.44,55804,1288237198579-1] regionserver.Store(412): closed family 2010-10-27 20:40:50,625 INFO [RS_CLOSE_REGION-192.168.0.44,55804,1288237198579-1] regionserver.HRegion(531): Closed tableRestart,xwp\xC4\xC1,1288237108306.4bdfd9bb922df7bfd8a1e1699ded4b04. 2010-10-27 20:40:50,625 DEBUG [RS_CLOSE_REGION-192.168.0.44,55804,1288237198579-1] handler.CloseRegionHandler(128): Closed region tableRestart,xwp\xC4\xC1,1288237108306.4bdfd9bb922df7bfd8a1e1699ded4b04. 2010-10-27 20:40:50,567 DEBUG [RS_CLOSE_REGION-192.168.0.44,55804,1288237198579-2] handler.CloseRegionHandler(128): Closed region tableRestart,q_\xEE\x8D!,1288237107203.84ef94e4fde0c75a0e3e39b2772d0155. 2010-10-27 20:40:50,625 INFO [RS_CLOSE_REGION-192.168.0.44,55804,1288237198579-0] regionserver.HRegion(531): Closed tableRestart,a\x8Er:\xD1,1288237103699.e8c41668483c8d9ae10681b14be0b2c3. 2010-10-27 20:40:50,582 DEBUG [RegionServer:3;192.168.0.44,55814,1288237212395] wal.HLog(772): Moved 1 log files to /user/jgray/.oldlogs 2010-10-27 20:40:50,625 DEBUG [RS_CLOSE_REGION-192.168.0.44,55804,1288237198579-0] handler.CloseRegionHandler(128): Closed region tableRestart,a\x8Er:\xD1,1288237103699.e8c41668483c8d9ae10681b14be0b2c3. 2010-10-27 20:40:50,627 INFO [RegionServer:3;192.168.0.44,55814,1288237212395] regionserver.HRegionServer(607): stopping server at: 192.168.0.44,55814,1288237212395 2010-10-27 20:40:50,627 DEBUG [RegionServer:3;192.168.0.44,55814,1288237212395] catalog.CatalogTracker(143): Stopping catalog tracker org.apache.hadoop.hbase.catalog.CatalogTracker@3b5cfd86 2010-10-27 20:40:50,627 INFO [RegionServer:3;192.168.0.44,55814,1288237212395] regionserver.Leases(124): RegionServer:3;192.168.0.44,55814,1288237212395 closing leases 2010-10-27 20:40:50,627 INFO [RegionServer:3;192.168.0.44,55814,1288237212395] regionserver.Leases(131): RegionServer:3;192.168.0.44,55814,1288237212395 closed leases 2010-10-27 20:40:50,627 DEBUG [RegionServer:3;192.168.0.44,55814,1288237212395] client.HConnectionManager$HConnectionImplementation(1062): Closed zookeeper sessionid=0x12bf0e9fd63001b 2010-10-27 20:40:50,631 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(227): master:55776-0x12bf0e9fd630015 Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/192.168.0.44,55814,1288237212395 2010-10-27 20:40:50,631 INFO [main-EventThread] zookeeper.RegionServerTracker(71): RegionServer ephemeral node deleted, processing expiration [192.168.0.44,55814,1288237212395] 2010-10-27 20:40:50,631 DEBUG [main-EventThread] master.ServerManager(524): Added=192.168.0.44,55814,1288237212395 to dead servers, submitted shutdown handler to be executed, root=true, meta=false 2010-10-27 20:40:50,631 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(227): master:55776-0x12bf0e9fd630015 Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2010-10-27 20:40:50,632 INFO [RegionServer:3;192.168.0.44,55814,1288237212395] regionserver.HRegionServer(625): RegionServer:3;192.168.0.44,55814,1288237212395 exiting 2010-10-27 20:40:50,632 INFO [MASTER_META_SERVER_OPERATIONS-192.168.0.44:55776-0] handler.ServerShutdownHandler(93): Splitting logs for 192.168.0.44,55814,1288237212395 2010-10-27 20:40:50,633 INFO [Shutdown of DFS[DFSClient[clientName=DFSClient_hb_rs_192.168.0.44,55814,1288237212395_1288237212581, ugi=jgray.hrs.5,supergroup]]] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(238): Hook closing fs=DFS[DFSClient[clientName=DFSClient_hb_rs_192.168.0.44,55814,1288237212395_1288237212581, ugi=jgray.hrs.5,supergroup]] 2010-10-27 20:40:50,634 DEBUG [main] master.TestRollingRestart(299): Root server down 2010-10-27 20:40:50,634 INFO [main] hbase.LocalHBaseCluster(250): Waiting on serverName=192.168.0.44,55804,1288237198579, load=(requests=0, regions=480, usedHeap=207, maxHeap=2995) 2010-10-27 20:40:50,739 INFO [MASTER_META_SERVER_OPERATIONS-192.168.0.44:55776-0] catalog.RootLocationEditor(43): Unsetting ROOT region location in ZooKeeper 2010-10-27 20:40:50,743 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(227): hconnection-0x12bf0e9fd63001f Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/root-region-server 2010-10-27 20:40:50,745 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(227): hconnection-0x12bf0e9fd630010 Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/root-region-server 2010-10-27 20:40:50,746 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(227): hconnection-0x12bf0e9fd63000e Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/root-region-server 2010-10-27 20:40:50,746 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(227): hconnection-0x12bf0e9fd63001d Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/root-region-server 2010-10-27 20:40:50,747 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(227): hconnection-0x12bf0e9fd63000a Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/root-region-server 2010-10-27 20:40:50,747 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(227): hconnection-0x12bf0e9fd630019 Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/root-region-server 2010-10-27 20:40:50,748 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(227): hconnection-0x12bf0e9fd630012 Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/root-region-server 2010-10-27 20:40:50,748 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(227): hconnection-0x12bf0e9fd630011 Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/root-region-server 2010-10-27 20:40:50,748 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(227): hconnection-0x12bf0e9fd63000c Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/root-region-server 2010-10-27 20:40:50,749 DEBUG [main-EventThread] zookeeper.ZKUtil(198): hconnection-0x12bf0e9fd630019 Set watcher on existing znode /hbase/root-region-server 2010-10-27 20:40:50,749 DEBUG [main-EventThread] zookeeper.ZKUtil(198): hconnection-0x12bf0e9fd630011 Set watcher on existing znode /hbase/root-region-server 2010-10-27 20:40:50,749 DEBUG [main-EventThread] zookeeper.ZKUtil(198): hconnection-0x12bf0e9fd630010 Set watcher on existing znode /hbase/root-region-server 2010-10-27 20:40:50,749 DEBUG [main-EventThread] zookeeper.ZKUtil(198): hconnection-0x12bf0e9fd630012 Set watcher on existing znode /hbase/root-region-server 2010-10-27 20:40:50,750 DEBUG [main-EventThread] zookeeper.ZKUtil(198): hconnection-0x12bf0e9fd63000e Set watcher on existing znode /hbase/root-region-server 2010-10-27 20:40:50,750 DEBUG [main-EventThread] zookeeper.ZKUtil(198): hconnection-0x12bf0e9fd63000c Set watcher on existing znode /hbase/root-region-server 2010-10-27 20:40:50,750 DEBUG [main-EventThread] zookeeper.ZKUtil(198): hconnection-0x12bf0e9fd63000a Set watcher on existing znode /hbase/root-region-server 2010-10-27 20:40:50,755 DEBUG [main-EventThread] zookeeper.ZKUtil(198): hconnection-0x12bf0e9fd63001d Set watcher on existing znode /hbase/root-region-server 2010-10-27 20:40:50,755 DEBUG [main-EventThread] zookeeper.ZKUtil(198): hconnection-0x12bf0e9fd63001f Set watcher on existing znode /hbase/root-region-server 2010-10-27 20:40:50,755 DEBUG [RegionServer:3;192.168.0.44,55862,1288237244570-EventThread] zookeeper.ZooKeeperWatcher(227): regionserver:55862-0x12bf0e9fd630020 Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/root-region-server 2010-10-27 20:40:50,760 DEBUG [RegionServer:3;192.168.0.44,55862,1288237244570-EventThread] zookeeper.ZKUtil(198): regionserver:55862-0x12bf0e9fd630020 Set watcher on existing znode /hbase/root-region-server 2010-10-27 20:40:50,761 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(227): hconnection-0x12bf0e9fd63000d Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/root-region-server 2010-10-27 20:40:50,762 DEBUG [main-EventThread] zookeeper.ZKUtil(198): hconnection-0x12bf0e9fd63000d Set watcher on existing znode /hbase/root-region-server 2010-10-27 20:40:50,762 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(227): hconnection-0x12bf0e9fd63000f Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/root-region-server 2010-10-27 20:40:50,762 DEBUG [main-EventThread] zookeeper.ZKUtil(198): hconnection-0x12bf0e9fd63000f Set watcher on existing znode /hbase/root-region-server 2010-10-27 20:40:50,762 DEBUG [RegionServer:3;192.168.0.44,55804,1288237198579-EventThread] zookeeper.ZooKeeperWatcher(227): regionserver:55804-0x12bf0e9fd63001a Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/root-region-server 2010-10-27 20:40:50,763 DEBUG [RegionServer:3;192.168.0.44,55804,1288237198579-EventThread] zookeeper.ZKUtil(198): regionserver:55804-0x12bf0e9fd63001a Set watcher on existing znode /hbase/root-region-server 2010-10-27 20:40:50,763 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(227): master:55776-0x12bf0e9fd630015 Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/root-region-server 2010-10-27 20:40:50,763 DEBUG [main-EventThread] zookeeper.ZKUtil(198): master:55776-0x12bf0e9fd630015 Set watcher on existing znode /hbase/root-region-server 2010-10-27 20:40:50,763 DEBUG [Master:0;192.168.0.44:55776-EventThread] zookeeper.ZooKeeperWatcher(227): hconnection-0x12bf0e9fd630016 Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/root-region-server 2010-10-27 20:40:50,763 DEBUG [Master:0;192.168.0.44:55776-EventThread] zookeeper.ZKUtil(198): hconnection-0x12bf0e9fd630016 Set watcher on existing znode /hbase/root-region-server 2010-10-27 20:40:50,764 DEBUG [RegionServer:3;192.168.0.44,55851,1288237229731-EventThread] zookeeper.ZooKeeperWatcher(227): regionserver:55851-0x12bf0e9fd63001e Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/root-region-server 2010-10-27 20:40:50,764 DEBUG [RegionServer:3;192.168.0.44,55851,1288237229731-EventThread] zookeeper.ZKUtil(198): regionserver:55851-0x12bf0e9fd63001e Set watcher on existing znode /hbase/root-region-server 2010-10-27 20:40:50,764 DEBUG [MASTER_META_SERVER_OPERATIONS-192.168.0.44:55776-0] zookeeper.ZKAssign(236): master:55776-0x12bf0e9fd630015 Creating (or updating) unassigned node for 70236052 with OFFLINE state 2010-10-27 20:40:50,766 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(227): master:55776-0x12bf0e9fd630015 Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/unassigned 2010-10-27 20:40:50,767 DEBUG [MASTER_META_SERVER_OPERATIONS-192.168.0.44:55776-0] master.AssignmentManager(907): No previous transition plan for -ROOT-,,0.70236052 so generated a random one; hri=-ROOT-,,0.70236052, src=, dest=192.168.0.44,55804,1288237198579; 3 (online=3, exclude=null) available servers 2010-10-27 20:40:50,767 DEBUG [MASTER_META_SERVER_OPERATIONS-192.168.0.44:55776-0] master.AssignmentManager(804): Assigning region -ROOT-,,0.70236052 to 192.168.0.44,55804,1288237198579 2010-10-27 20:40:50,768 WARN [MASTER_META_SERVER_OPERATIONS-192.168.0.44:55776-0] master.AssignmentManager(811): Failed assignment of -ROOT-,,0.70236052 to serverName=192.168.0.44,55804,1288237198579, load=(requests=0, regions=480, usedHeap=207, maxHeap=2995), trying to assign elsewhere instead java.io.IOException: Call to /192.168.0.44:55804 failed on local exception: java.io.EOFException at org.apache.hadoop.hbase.ipc.HBaseClient.wrapException(HBaseClient.java:784) at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:753) at org.apache.hadoop.hbase.ipc.HBaseRPC$Invoker.invoke(HBaseRPC.java:256) at $Proxy8.openRegion(Unknown Source) at org.apache.hadoop.hbase.master.ServerManager.sendRegionOpen(ServerManager.java:547) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:809) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:643) at org.apache.hadoop.hbase.master.AssignmentManager.assignRoot(AssignmentManager.java:1035) at org.apache.hadoop.hbase.master.handler.ServerShutdownHandler.process(ServerShutdownHandler.java:105) at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:151) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:680) Caused by: java.io.EOFException at java.io.DataInputStream.readInt(DataInputStream.java:375) at org.apache.hadoop.hbase.ipc.HBaseClient$Connection.receiveResponse(HBaseClient.java:517) at org.apache.hadoop.hbase.ipc.HBaseClient$Connection.run(HBaseClient.java:455) 2010-10-27 20:40:50,771 DEBUG [MASTER_META_SERVER_OPERATIONS-192.168.0.44:55776-0] master.AssignmentManager(907): No previous transition plan for -ROOT-,,0.70236052 so generated a random one; hri=-ROOT-,,0.70236052, src=, dest=192.168.0.44,55862,1288237244570; 3 (online=3, exclude=serverName=192.168.0.44,55804,1288237198579, load=(requests=0, regions=480, usedHeap=207, maxHeap=2995)) available servers 2010-10-27 20:40:50,771 FATAL [MASTER_META_SERVER_OPERATIONS-192.168.0.44:55776-0] master.HMaster(874): Unexpected state trying to OFFLINE; -ROOT-,,0.70236052 state=PENDING_OPEN, ts=1288237250767 java.lang.IllegalStateException at org.apache.hadoop.hbase.master.AssignmentManager.setOfflineInZooKeeper(AssignmentManager.java:834) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:796) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:820) at org.apache.hadoop.hbase.master.AssignmentManager.assign(AssignmentManager.java:643) at org.apache.hadoop.hbase.master.AssignmentManager.assignRoot(AssignmentManager.java:1035) at org.apache.hadoop.hbase.master.handler.ServerShutdownHandler.process(ServerShutdownHandler.java:105) at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:151) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:680) 2010-10-27 20:40:50,771 INFO [MASTER_META_SERVER_OPERATIONS-192.168.0.44:55776-0] master.HMaster(917): Aborting