2010-10-27 11:47:26,292 DEBUG [MASTER_META_SERVER_OPERATIONS-172.24.154.204:49403-1] zookeeper.ZKAssign(236): master:49403-0x12bef034d190015 Creating (or updating) unassigned node for 1028785192 with OFFLINE state 2010-10-27 11:47:26,294 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(227): master:49403-0x12bef034d190015 Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/unassigned/1028785192 2010-10-27 11:47:26,294 DEBUG [main-EventThread] zookeeper.ZKUtil(1058): master:49403-0x12bef034d190015 Retrieved 43 byte(s) of data from znode /hbase/unassigned/1028785192 and set watcher; U .META.,,1 2010-10-27 11:47:26,294 DEBUG [MASTER_META_SERVER_OPERATIONS-172.24.154.204:49403-1] master.AssignmentManager(906): No previous transition plan for .META.,,1.1028785192 so generated a random one; hri=.META.,,1.1028785192, src=, dest=172.24.154.204,49531,1288205239169; 1 (online=1, exclude=null) available servers 2010-10-27 11:47:26,294 DEBUG [MASTER_META_SERVER_OPERATIONS-172.24.154.204:49403-1] master.AssignmentManager(803): Assigning region .META.,,1.1028785192 to 172.24.154.204,49531,1288205239169 2010-10-27 11:47:26,295 DEBUG [main-EventThread] zookeeper.ZKUtil(1058): master:49403-0x12bef034d190015 Retrieved 43 byte(s) of data from znode /hbase/unassigned/1028785192 and set watcher; U .META.,,1 2010-10-27 11:47:26,295 DEBUG [main-EventThread] master.AssignmentManager(324): Handling transition=M_ZK_REGION_OFFLINE, server=172.24.154.204:49403, region=1028785192/.META. 2010-10-27 11:47:26,295 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(227): master:49403-0x12bef034d190015 Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/unassigned 2010-10-27 11:47:26,298 INFO [MASTER_META_SERVER_OPERATIONS-172.24.154.204:49403-1] catalog.CatalogTracker(436): Failed verification of .META.,,1 at address=172.24.154.204:49536; java.io.IOException: Connection reset by peer 2010-10-27 11:47:26,299 INFO [MASTER_META_SERVER_OPERATIONS-172.24.154.204:49403-1] catalog.CatalogTracker(369): Current cached META location is not valid, resetting 2010-10-27 11:47:26,301 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(227): master:49403-0x12bef034d190015 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/unassigned/1028785192 2010-10-27 11:47:26,301 INFO [MASTER_META_SERVER_OPERATIONS-172.24.154.204:49403-1] catalog.CatalogTracker(436): Failed verification of .META.,,1 at address=172.24.154.204:49536; java.net.ConnectException: Connection refused 2010-10-27 11:47:26,301 DEBUG [main] master.TestRollingRestart(275): 2010-10-27 11:47:26,304 DEBUG [main-EventThread] zookeeper.ZKUtil(1058): master:49403-0x12bef034d190015 Retrieved 57 byte(s) of data from znode /hbase/unassigned/1028785192 and set watcher; Z .META.,,1 2010-10-27 11:47:26,305 DEBUG [main-EventThread] zookeeper.ZKUtil(1058): master:49403-0x12bef034d190015 Retrieved 57 byte(s) of data from znode /hbase/unassigned/1028785192 and set watcher; Z .META.,,1 2010-10-27 11:47:26,306 DEBUG [main-EventThread] master.AssignmentManager(324): Handling transition=RS_ZK_REGION_OPENING, server=172.24.154.204,49531,1288205239169, region=1028785192/.META. 2010-10-27 11:47:26,342 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(227): master:49403-0x12bef034d190015 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/unassigned/1028785192 2010-10-27 11:47:26,346 DEBUG [main-EventThread] zookeeper.ZKUtil(1058): master:49403-0x12bef034d190015 Retrieved 57 byte(s) of data from znode /hbase/unassigned/1028785192 and set watcher;  .META.,,1 2010-10-27 11:47:26,347 DEBUG [main-EventThread] zookeeper.ZKUtil(1058): master:49403-0x12bef034d190015 Retrieved 57 byte(s) of data from znode /hbase/unassigned/1028785192 and set watcher;  .META.,,1 2010-10-27 11:47:26,347 DEBUG [main-EventThread] master.AssignmentManager(324): Handling transition=RS_ZK_REGION_OPENED, server=172.24.154.204,49531,1288205239169, region=1028785192/.META. 2010-10-27 11:47:26,347 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(227): master:49403-0x12bef034d190015 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/unassigned/1028785192 2010-10-27 11:47:26,347 DEBUG [main-EventThread] zookeeper.ZKUtil(1058): master:49403-0x12bef034d190015 Retrieved 57 byte(s) of data from znode /hbase/unassigned/1028785192 and set watcher;  .META.,,1 2010-10-27 11:47:26,348 DEBUG [main-EventThread] zookeeper.ZKUtil(1058): master:49403-0x12bef034d190015 Retrieved 57 byte(s) of data from znode /hbase/unassigned/1028785192 and set watcher;  .META.,,1 2010-10-27 11:47:26,348 DEBUG [main-EventThread] master.AssignmentManager(324): Handling transition=RS_ZK_REGION_OPENED, server=172.24.154.204,49531,1288205239169, region=1028785192/.META. 2010-10-27 11:47:26,348 DEBUG [MASTER_OPEN_REGION-172.24.154.204:49403-3] handler.OpenedRegionHandler(85): Handling OPENED event for 1028785192; deleting unassigned node 2010-10-27 11:47:26,348 DEBUG [MASTER_OPEN_REGION-172.24.154.204:49403-3] zookeeper.ZKAssign(392): master:49403-0x12bef034d190015 Deleting existing unassigned node for 1028785192 that is in expected state RS_ZK_REGION_OPENED 2010-10-27 11:47:26,348 DEBUG [MASTER_OPEN_REGION-172.24.154.204:49403-1] handler.OpenedRegionHandler(85): Handling OPENED event for 1028785192; deleting unassigned node 2010-10-27 11:47:26,348 DEBUG [MASTER_OPEN_REGION-172.24.154.204:49403-1] zookeeper.ZKAssign(392): master:49403-0x12bef034d190015 Deleting existing unassigned node for 1028785192 that is in expected state RS_ZK_REGION_OPENED 2010-10-27 11:47:26,349 DEBUG [MASTER_OPEN_REGION-172.24.154.204:49403-3] zookeeper.ZKUtil(1058): master:49403-0x12bef034d190015 Retrieved 57 byte(s) of data from znode /hbase/unassigned/1028785192; data= .META.,,1 2010-10-27 11:47:26,349 DEBUG [MASTER_OPEN_REGION-172.24.154.204:49403-1] zookeeper.ZKUtil(1058): master:49403-0x12bef034d190015 Retrieved 57 byte(s) of data from znode /hbase/unassigned/1028785192; data= .META.,,1 2010-10-27 11:47:26,349 DEBUG [main] master.TestRollingRestart(275): Server [172.24.154.204,49536,1288205240186] still marked as dead, waiting 2010-10-27 11:47:26,350 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(227): master:49403-0x12bef034d190015 Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/unassigned/1028785192 2010-10-27 11:47:26,350 DEBUG [MASTER_OPEN_REGION-172.24.154.204:49403-3] zookeeper.ZKAssign(418): master:49403-0x12bef034d190015 Successfully deleted unassigned node for region 1028785192 in expected state RS_ZK_REGION_OPENED 2010-10-27 11:47:26,350 DEBUG [MASTER_OPEN_REGION-172.24.154.204:49403-3] handler.OpenedRegionHandler(102): Opened region .META.,,1.1028785192 2010-10-27 11:47:26,351 DEBUG [main-EventThread] zookeeper.ZKUtil(198): master:49403-0x12bef034d190015 Set watcher on existing znode /hbase/unassigned/1028785192 2010-10-27 11:47:26,351 INFO [main-EventThread] zookeeper.MetaNodeTracker(62): Detected completed assignment of META, notifying catalog tracker 2010-10-27 11:47:26,352 FATAL [MASTER_OPEN_REGION-172.24.154.204:49403-1] master.HMaster(874): Error deleting OPENED node in ZK for transition ZK node (1028785192) org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /hbase/unassigned/1028785192 at org.apache.zookeeper.KeeperException.create(KeeperException.java:102) at org.apache.zookeeper.KeeperException.create(KeeperException.java:42) at org.apache.zookeeper.ZooKeeper.delete(ZooKeeper.java:728) at org.apache.hadoop.hbase.zookeeper.ZKUtil.deleteNode(ZKUtil.java:909) at org.apache.hadoop.hbase.zookeeper.ZKAssign.deleteNode(ZKAssign.java:411) at org.apache.hadoop.hbase.zookeeper.ZKAssign.deleteOpenedNode(ZKAssign.java:288) at org.apache.hadoop.hbase.master.handler.OpenedRegionHandler.process(OpenedRegionHandler.java:89) at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:151) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:680) 2010-10-27 11:47:26,352 INFO [MASTER_OPEN_REGION-172.24.154.204:49403-1] master.HMaster(917): Aborting