2010-10-27 11:47:26,292 DEBUG [MASTER_META_SERVER_OPERATIONS-172.24.154.204:49403-1] zookeeper.ZKAssign(236): master:49403-0x12bef034d190015 Creating (or updating) unassigned node for 1028785192 with OFFLINE state 2010-10-27 11:47:26,294 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(227): master:49403-0x12bef034d190015 Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/unassigned/1028785192 2010-10-27 11:47:26,294 DEBUG [main-EventThread] zookeeper.ZKUtil(1058): master:49403-0x12bef034d190015 Retrieved 43 byte(s) of data from znode /hbase/unassigned/1028785192 and set watcher; U .META.,,1 2010-10-27 11:47:26,294 DEBUG [MASTER_META_SERVER_OPERATIONS-172.24.154.204:49403-1] master.AssignmentManager(906): No previous transition plan for .META.,,1.1028785192 so generated a random one; hri=.META.,,1.1028785192, src=, dest=172.24.154.204,49531,1288205239169; 1 (online=1, exclude=null) available servers 2010-10-27 11:47:26,294 DEBUG [MASTER_META_SERVER_OPERATIONS-172.24.154.204:49403-1] master.AssignmentManager(803): Assigning region .META.,,1.1028785192 to 172.24.154.204,49531,1288205239169 2010-10-27 11:47:26,295 DEBUG [main-EventThread] zookeeper.ZKUtil(1058): master:49403-0x12bef034d190015 Retrieved 43 byte(s) of data from znode /hbase/unassigned/1028785192 and set watcher; U .META.,,1 2010-10-27 11:47:26,295 DEBUG [main-EventThread] master.AssignmentManager(324): Handling transition=M_ZK_REGION_OFFLINE, server=172.24.154.204:49403, region=1028785192/.META. 2010-10-27 11:47:26,295 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(227): master:49403-0x12bef034d190015 Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/unassigned 2010-10-27 11:47:26,295 DEBUG [RegionServer:2;172.24.154.204,49531,1288205239169-EventThread] zookeeper.ZooKeeperWatcher(227): regionserver:49531-0x12bef034d190024 Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/unassigned/1028785192 2010-10-27 11:47:26,296 DEBUG [RegionServer:2;172.24.154.204,49531,1288205239169-EventThread] zookeeper.ZKUtil(1058): regionserver:49531-0x12bef034d190024 Retrieved 43 byte(s) of data from znode /hbase/unassigned/1028785192 and set watcher; U .META.,,1 2010-10-27 11:47:26,296 INFO [IPC Server handler 4 on 49531] regionserver.HRegionServer(1952): Received request to open region: .META.,,1.1028785192 2010-10-27 11:47:26,297 DEBUG [RS_OPEN_META-172.24.154.204,49531,1288205239169-0] handler.OpenRegionHandler(69): Processing open of .META.,,1.1028785192 2010-10-27 11:47:26,297 DEBUG [RS_OPEN_META-172.24.154.204,49531,1288205239169-0] zookeeper.ZKAssign(657): regionserver:49531-0x12bef034d190024 Attempting to transition node 1028785192/.META. from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING 2010-10-27 11:47:26,298 DEBUG [RS_OPEN_META-172.24.154.204,49531,1288205239169-0] zookeeper.ZKUtil(1058): regionserver:49531-0x12bef034d190024 Retrieved 43 byte(s) of data from znode /hbase/unassigned/1028785192; data=U .META.,,1 2010-10-27 11:47:26,298 INFO [MASTER_META_SERVER_OPERATIONS-172.24.154.204:49403-1] catalog.CatalogTracker(436): Failed verification of .META.,,1 at address=172.24.154.204:49536; java.io.IOException: Connection reset by peer 2010-10-27 11:47:26,299 INFO [MASTER_META_SERVER_OPERATIONS-172.24.154.204:49403-1] catalog.CatalogTracker(369): Current cached META location is not valid, resetting 2010-10-27 11:47:26,300 DEBUG [RegionServer:2;172.24.154.204,49531,1288205239169-EventThread] zookeeper.ZooKeeperWatcher(227): regionserver:49531-0x12bef034d190024 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/unassigned/1028785192 2010-10-27 11:47:26,300 DEBUG [RS_OPEN_META-172.24.154.204,49531,1288205239169-0] zookeeper.ZKAssign(704): regionserver:49531-0x12bef034d190024 Successfully transitioned node 1028785192 from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING 2010-10-27 11:47:26,300 DEBUG [RS_OPEN_META-172.24.154.204,49531,1288205239169-0] regionserver.HRegion(2448): Opening region: REGION => {NAME => '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192, TABLE => {{NAME => '.META.', IS_META => 'true', FAMILIES => [{NAME => 'info', BLOOMFILTER => 'NONE', REPLICATION_SCOPE => '0', VERSIONS => '10', COMPRESSION => 'NONE', TTL => '2147483647', BLOCKSIZE => '8192', IN_MEMORY => 'true', BLOCKCACHE => 'true'}]}} 2010-10-27 11:47:26,300 DEBUG [RS_OPEN_META-172.24.154.204,49531,1288205239169-0] regionserver.HRegion(302): Instantiated .META.,,1.1028785192 2010-10-27 11:47:26,301 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(227): master:49403-0x12bef034d190015 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/unassigned/1028785192 2010-10-27 11:47:26,301 INFO [MASTER_META_SERVER_OPERATIONS-172.24.154.204:49403-1] catalog.CatalogTracker(436): Failed verification of .META.,,1 at address=172.24.154.204:49536; java.net.ConnectException: Connection refused 2010-10-27 11:47:26,301 DEBUG [main] master.TestRollingRestart(275): Server [172.24.154.204,49536,1288205240186] still marked as dead, waiting 2010-10-27 11:47:26,304 DEBUG [RegionServer:2;172.24.154.204,49531,1288205239169-EventThread] zookeeper.ZKUtil(1058): regionserver:49531-0x12bef034d190024 Retrieved 57 byte(s) of data from znode /hbase/unassigned/1028785192 and set watcher; Z .META.,,1 2010-10-27 11:47:26,304 DEBUG [main-EventThread] zookeeper.ZKUtil(1058): master:49403-0x12bef034d190015 Retrieved 57 byte(s) of data from znode /hbase/unassigned/1028785192 and set watcher; Z .META.,,1 2010-10-27 11:47:26,305 DEBUG [main-EventThread] zookeeper.ZKUtil(1058): master:49403-0x12bef034d190015 Retrieved 57 byte(s) of data from znode /hbase/unassigned/1028785192 and set watcher; Z .META.,,1 2010-10-27 11:47:26,306 DEBUG [main-EventThread] master.AssignmentManager(324): Handling transition=RS_ZK_REGION_OPENING, server=172.24.154.204,49531,1288205239169, region=1028785192/.META. 2010-10-27 11:47:26,317 DEBUG [main] master.TestRollingRestart(275): Server [172.24.154.204,49536,1288205240186] still marked as dead, waiting 2010-10-27 11:47:26,330 DEBUG [RS_OPEN_META-172.24.154.204,49531,1288205239169-0] regionserver.Store(270): loaded hdfs://localhost:49236/user/jgray/.META./1028785192/info/1573136359505371624, isReference=false, isBulkLoadResult=false, seqid=198, majorCompaction=false 2010-10-27 11:47:26,334 DEBUG [RS_OPEN_META-172.24.154.204,49531,1288205239169-0] regionserver.Store(270): loaded hdfs://localhost:49236/user/jgray/.META./1028785192/info/5063767091724875597, isReference=false, isBulkLoadResult=false, seqid=209, majorCompaction=false 2010-10-27 11:47:26,337 DEBUG [main] master.TestRollingRestart(275): Server [172.24.154.204,49536,1288205240186] still marked as dead, waiting 2010-10-27 11:47:26,338 DEBUG [RS_OPEN_META-172.24.154.204,49531,1288205239169-0] regionserver.Store(270): loaded hdfs://localhost:49236/user/jgray/.META./1028785192/info/5432631253454732461, isReference=false, isBulkLoadResult=false, seqid=170, majorCompaction=false 2010-10-27 11:47:26,340 INFO [RS_OPEN_META-172.24.154.204,49531,1288205239169-0] regionserver.HRegion(363): Onlined .META.,,1.1028785192; next sequenceid=210 2010-10-27 11:47:26,340 DEBUG [RS_OPEN_META-172.24.154.204,49531,1288205239169-0] zookeeper.ZKAssign(657): regionserver:49531-0x12bef034d190024 Attempting to transition node 1028785192/.META. from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENING 2010-10-27 11:47:26,341 DEBUG [RS_OPEN_META-172.24.154.204,49531,1288205239169-0] zookeeper.ZKUtil(1058): regionserver:49531-0x12bef034d190024 Retrieved 57 byte(s) of data from znode /hbase/unassigned/1028785192; data=Z .META.,,1 2010-10-27 11:47:26,341 DEBUG [RegionServer:2;172.24.154.204,49531,1288205239169-EventThread] zookeeper.ZooKeeperWatcher(227): regionserver:49531-0x12bef034d190024 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/unassigned/1028785192 2010-10-27 11:47:26,342 DEBUG [RS_OPEN_META-172.24.154.204,49531,1288205239169-0] zookeeper.ZKAssign(704): regionserver:49531-0x12bef034d190024 Successfully transitioned node 1028785192 from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENING 2010-10-27 11:47:26,342 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(227): master:49403-0x12bef034d190015 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/unassigned/1028785192 2010-10-27 11:47:26,344 INFO [RS_OPEN_META-172.24.154.204,49531,1288205239169-0] catalog.MetaEditor(169): Updated row .META.,,1.1028785192 in region -ROOT-,,0 with server=172.24.154.204:49531, startcode=1288205239169 2010-10-27 11:47:26,345 DEBUG [RS_OPEN_META-172.24.154.204,49531,1288205239169-0] zookeeper.ZKAssign(657): regionserver:49531-0x12bef034d190024 Attempting to transition node 1028785192/.META. from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENED 2010-10-27 11:47:26,345 DEBUG [RegionServer:2;172.24.154.204,49531,1288205239169-EventThread] zookeeper.ZKUtil(1058): regionserver:49531-0x12bef034d190024 Retrieved 57 byte(s) of data from znode /hbase/unassigned/1028785192 and set watcher;  .META.,,1 2010-10-27 11:47:26,345 DEBUG [RS_OPEN_META-172.24.154.204,49531,1288205239169-0] zookeeper.ZKUtil(1058): regionserver:49531-0x12bef034d190024 Retrieved 57 byte(s) of data from znode /hbase/unassigned/1028785192; data= .META.,,1 2010-10-27 11:47:26,346 DEBUG [main-EventThread] zookeeper.ZKUtil(1058): master:49403-0x12bef034d190015 Retrieved 57 byte(s) of data from znode /hbase/unassigned/1028785192 and set watcher;  .META.,,1 2010-10-27 11:47:26,346 DEBUG [RegionServer:2;172.24.154.204,49531,1288205239169-EventThread] zookeeper.ZooKeeperWatcher(227): regionserver:49531-0x12bef034d190024 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/unassigned/1028785192 2010-10-27 11:47:26,346 DEBUG [RS_OPEN_META-172.24.154.204,49531,1288205239169-0] zookeeper.ZKAssign(704): regionserver:49531-0x12bef034d190024 Successfully transitioned node 1028785192 from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENED 2010-10-27 11:47:26,346 DEBUG [RS_OPEN_META-172.24.154.204,49531,1288205239169-0] handler.OpenRegionHandler(195): Opened .META.,,1.1028785192 2010-10-27 11:47:26,347 DEBUG [main-EventThread] zookeeper.ZKUtil(1058): master:49403-0x12bef034d190015 Retrieved 57 byte(s) of data from znode /hbase/unassigned/1028785192 and set watcher;  .META.,,1 2010-10-27 11:47:26,347 DEBUG [main-EventThread] master.AssignmentManager(324): Handling transition=RS_ZK_REGION_OPENED, server=172.24.154.204,49531,1288205239169, region=1028785192/.META. 2010-10-27 11:47:26,347 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(227): master:49403-0x12bef034d190015 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/unassigned/1028785192 2010-10-27 11:47:26,347 DEBUG [RegionServer:2;172.24.154.204,49531,1288205239169-EventThread] zookeeper.ZKUtil(1058): regionserver:49531-0x12bef034d190024 Retrieved 57 byte(s) of data from znode /hbase/unassigned/1028785192 and set watcher;  .META.,,1 2010-10-27 11:47:26,347 DEBUG [main-EventThread] zookeeper.ZKUtil(1058): master:49403-0x12bef034d190015 Retrieved 57 byte(s) of data from znode /hbase/unassigned/1028785192 and set watcher;  .META.,,1 2010-10-27 11:47:26,348 DEBUG [main-EventThread] zookeeper.ZKUtil(1058): master:49403-0x12bef034d190015 Retrieved 57 byte(s) of data from znode /hbase/unassigned/1028785192 and set watcher;  .META.,,1 2010-10-27 11:47:26,348 DEBUG [main-EventThread] master.AssignmentManager(324): Handling transition=RS_ZK_REGION_OPENED, server=172.24.154.204,49531,1288205239169, region=1028785192/.META. 2010-10-27 11:47:26,348 DEBUG [MASTER_OPEN_REGION-172.24.154.204:49403-3] handler.OpenedRegionHandler(85): Handling OPENED event for 1028785192; deleting unassigned node 2010-10-27 11:47:26,348 DEBUG [MASTER_OPEN_REGION-172.24.154.204:49403-3] zookeeper.ZKAssign(392): master:49403-0x12bef034d190015 Deleting existing unassigned node for 1028785192 that is in expected state RS_ZK_REGION_OPENED 2010-10-27 11:47:26,348 DEBUG [MASTER_OPEN_REGION-172.24.154.204:49403-1] handler.OpenedRegionHandler(85): Handling OPENED event for 1028785192; deleting unassigned node 2010-10-27 11:47:26,348 DEBUG [MASTER_OPEN_REGION-172.24.154.204:49403-1] zookeeper.ZKAssign(392): master:49403-0x12bef034d190015 Deleting existing unassigned node for 1028785192 that is in expected state RS_ZK_REGION_OPENED 2010-10-27 11:47:26,349 DEBUG [MASTER_OPEN_REGION-172.24.154.204:49403-3] zookeeper.ZKUtil(1058): master:49403-0x12bef034d190015 Retrieved 57 byte(s) of data from znode /hbase/unassigned/1028785192; data= .META.,,1 2010-10-27 11:47:26,349 DEBUG [MASTER_OPEN_REGION-172.24.154.204:49403-1] zookeeper.ZKUtil(1058): master:49403-0x12bef034d190015 Retrieved 57 byte(s) of data from znode /hbase/unassigned/1028785192; data= .META.,,1 2010-10-27 11:47:26,349 DEBUG [main] master.TestRollingRestart(275): Server [172.24.154.204,49536,1288205240186] still marked as dead, waiting 2010-10-27 11:47:26,350 DEBUG [RegionServer:2;172.24.154.204,49531,1288205239169-EventThread] zookeeper.ZooKeeperWatcher(227): regionserver:49531-0x12bef034d190024 Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/unassigned/1028785192 2010-10-27 11:47:26,350 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(227): master:49403-0x12bef034d190015 Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/unassigned/1028785192 2010-10-27 11:47:26,350 DEBUG [MASTER_OPEN_REGION-172.24.154.204:49403-3] zookeeper.ZKAssign(418): master:49403-0x12bef034d190015 Successfully deleted unassigned node for region 1028785192 in expected state RS_ZK_REGION_OPENED 2010-10-27 11:47:26,350 DEBUG [MASTER_OPEN_REGION-172.24.154.204:49403-3] handler.OpenedRegionHandler(102): Opened region .META.,,1.1028785192 2010-10-27 11:47:26,351 DEBUG [main-EventThread] zookeeper.ZKUtil(198): master:49403-0x12bef034d190015 Set watcher on existing znode /hbase/unassigned/1028785192 2010-10-27 11:47:26,351 INFO [main-EventThread] zookeeper.MetaNodeTracker(62): Detected completed assignment of META, notifying catalog tracker 2010-10-27 11:47:26,352 FATAL [MASTER_OPEN_REGION-172.24.154.204:49403-1] master.HMaster(874): Error deleting OPENED node in ZK for transition ZK node (1028785192) org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /hbase/unassigned/1028785192 at org.apache.zookeeper.KeeperException.create(KeeperException.java:102) at org.apache.zookeeper.KeeperException.create(KeeperException.java:42) at org.apache.zookeeper.ZooKeeper.delete(ZooKeeper.java:728) at org.apache.hadoop.hbase.zookeeper.ZKUtil.deleteNode(ZKUtil.java:909) at org.apache.hadoop.hbase.zookeeper.ZKAssign.deleteNode(ZKAssign.java:411) at org.apache.hadoop.hbase.zookeeper.ZKAssign.deleteOpenedNode(ZKAssign.java:288) at org.apache.hadoop.hbase.master.handler.OpenedRegionHandler.process(OpenedRegionHandler.java:89) at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:151) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:680) 2010-10-27 11:47:26,352 INFO [MASTER_OPEN_REGION-172.24.154.204:49403-1] master.HMaster(917): Aborting