2013-04-07 12:14:34,786 INFO [pool-1-thread-1] hbase.HBaseTestingUtility(354): Created new mini-cluster data directory: /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/trunk/hbase-server/target/test-data/4074d7d7-ccf0-4faa-9140-f2fccb960394/dfscluster_3f48034d-4746-4e9c-930e-50470ccdf51e 2013-04-07 12:14:34,958 INFO [pool-1-thread-1] zookeeper.MiniZooKeeperCluster(197): Started MiniZK Cluster and connect 1 ZK server on client port: 51597 2013-04-07 12:14:34,959 INFO [pool-1-thread-1] hbase.HBaseTestingUtility(776): Starting up minicluster with 1 master(s) and 2 regionserver(s) and 2 datanode(s) 2013-04-07 12:14:34,960 INFO [pool-1-thread-1] hbase.HBaseTestingUtility(584): Setting test.cache.data to /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/trunk/hbase-server/target/test-data/4074d7d7-ccf0-4faa-9140-f2fccb960394/cache_data in system properties and HBase conf 2013-04-07 12:14:34,960 INFO [pool-1-thread-1] hbase.HBaseTestingUtility(584): Setting hadoop.tmp.dir to /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/trunk/hbase-server/target/test-data/4074d7d7-ccf0-4faa-9140-f2fccb960394/hadoop_tmp in system properties and HBase conf 2013-04-07 12:14:34,961 INFO [pool-1-thread-1] hbase.HBaseTestingUtility(584): Setting hadoop.log.dir to /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/trunk/hbase-server/target/test-data/4074d7d7-ccf0-4faa-9140-f2fccb960394/hadoop_logs in system properties and HBase conf 2013-04-07 12:14:34,961 INFO [pool-1-thread-1] hbase.HBaseTestingUtility(584): Setting mapred.local.dir to /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/trunk/hbase-server/target/test-data/4074d7d7-ccf0-4faa-9140-f2fccb960394/mapred_local in system properties and HBase conf 2013-04-07 12:14:34,962 INFO [pool-1-thread-1] hbase.HBaseTestingUtility(584): Setting mapred.temp.dir to /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/trunk/hbase-server/target/test-data/4074d7d7-ccf0-4faa-9140-f2fccb960394/mapred_temp in system properties and HBase conf 2013-04-07 12:14:34,962 INFO [pool-1-thread-1] hbase.HBaseTestingUtility(567): read short circuit is ON for user jenkins 2013-04-07 12:14:35,152 DEBUG [pool-1-thread-1] fs.HFileSystem(199): Starting addLocationsOrderInterceptor with class class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2013-04-07 12:14:35,154 WARN [pool-1-thread-1] fs.HFileSystem(215): The file system is not a DistributedFileSystem.Not adding block location reordering 2013-04-07 12:14:35,322 WARN [pool-1-thread-1] namenode.FSNamesystem(564): The dfs.support.append option is in your configuration, however append is not supported. This configuration option is no longer required to enable sync. 2013-04-07 12:14:35,936 WARN [pool-1-thread-1] namenode.FSNamesystem(564): The dfs.support.append option is in your configuration, however append is not supported. This configuration option is no longer required to enable sync. 2013-04-07 12:14:36,320 INFO [pool-1-thread-1] log.Slf4jLog(67): Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog 2013-04-07 12:14:36,404 INFO [pool-1-thread-1] log.Slf4jLog(67): jetty-6.1.26 2013-04-07 12:14:36,465 INFO [pool-1-thread-1] log.Slf4jLog(67): Extract jar:file:/home/jenkins/.m2/repository/org/apache/hadoop/hadoop-core/1.1.2/hadoop-core-1.1.2.jar!/webapps/hdfs to /tmp/Jetty_localhost_41391_hdfs____.ukr8tk/webapp 2013-04-07 12:14:36,794 INFO [pool-1-thread-1] log.Slf4jLog(67): Started SelectChannelConnector@localhost:41391 Starting DataNode 0 with dfs.data.dir: /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/trunk/hbase-server/target/test-data/4074d7d7-ccf0-4faa-9140-f2fccb960394/dfscluster_3f48034d-4746-4e9c-930e-50470ccdf51e/dfs/data/data1,/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/trunk/hbase-server/target/test-data/4074d7d7-ccf0-4faa-9140-f2fccb960394/dfscluster_3f48034d-4746-4e9c-930e-50470ccdf51e/dfs/data/data2 2013-04-07 12:14:37,022 INFO [pool-1-thread-1] log.Slf4jLog(67): jetty-6.1.26 2013-04-07 12:14:37,037 INFO [pool-1-thread-1] log.Slf4jLog(67): Extract jar:file:/home/jenkins/.m2/repository/org/apache/hadoop/hadoop-core/1.1.2/hadoop-core-1.1.2.jar!/webapps/datanode to /tmp/Jetty_localhost_48590_datanode____.4ztqtt/webapp 2013-04-07 12:14:37,237 INFO [pool-1-thread-1] log.Slf4jLog(67): Started SelectChannelConnector@localhost:48590 Starting DataNode 1 with dfs.data.dir: /home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/trunk/hbase-server/target/test-data/4074d7d7-ccf0-4faa-9140-f2fccb960394/dfscluster_3f48034d-4746-4e9c-930e-50470ccdf51e/dfs/data/data3,/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/trunk/hbase-server/target/test-data/4074d7d7-ccf0-4faa-9140-f2fccb960394/dfscluster_3f48034d-4746-4e9c-930e-50470ccdf51e/dfs/data/data4 2013-04-07 12:14:37,366 INFO [pool-1-thread-1] log.Slf4jLog(67): jetty-6.1.26 2013-04-07 12:14:37,380 INFO [pool-1-thread-1] log.Slf4jLog(67): Extract jar:file:/home/jenkins/.m2/repository/org/apache/hadoop/hadoop-core/1.1.2/hadoop-core-1.1.2.jar!/webapps/datanode to /tmp/Jetty_localhost_43017_datanode____gih2am/webapp 2013-04-07 12:14:37,570 INFO [pool-1-thread-1] log.Slf4jLog(67): Started SelectChannelConnector@localhost:43017 Cluster is active 2013-04-07 12:14:37,928 DEBUG [pool-1-thread-1] util.FSUtils(511): Created version file at hdfs://localhost:60405/user/jenkins/hbase with version=7 2013-04-07 12:14:38,005 DEBUG [pool-1-thread-1] client.HConnectionManager(2673): HConnection retries=100 2013-04-07 12:14:38,047 INFO [pool-1-thread-1] master.HMaster(416): hbase.rootdir=hdfs://localhost:60405/user/jenkins/hbase, hbase.cluster.distributed=false 2013-04-07 12:14:38,052 DEBUG [pool-1-thread-1] zookeeper.ZKConfig(84): Skipped reading ZK properties file 'zoo.cfg' since 'hbase.config.read.zookeeper.config' was not set to true 2013-04-07 12:14:38,053 DEBUG [pool-1-thread-1] zookeeper.ZKUtil(119): master:33796 opening connection to ZooKeeper with ensemble (localhost:51597) 2013-04-07 12:14:38,075 INFO [pool-1-thread-1] zookeeper.RecoverableZooKeeper(119): The identifier of this process is master:33796 2013-04-07 12:14:38,120 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(274): master:33796 Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2013-04-07 12:14:38,124 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(351): master:33796-0x13de46b5f8b0000 connected 2013-04-07 12:14:38,285 DEBUG [pool-1-thread-1] client.HConnectionManager(2673): HConnection retries=100 2013-04-07 12:14:38,391 INFO [pool-1-thread-1] hfile.CacheConfig(410): Allocating LruBlockCache with maximum size 422.2m 2013-04-07 12:14:38,439 DEBUG [pool-1-thread-1] client.HConnectionManager(2673): HConnection retries=100 2013-04-07 12:14:38,453 DEBUG [Master:0;asf001.sp2.ygridcore.net,33796,1365336878036] zookeeper.ZKUtil(424): master:33796-0x13de46b5f8b0000 /hbase/running does not exist. Watcher is set. 2013-04-07 12:14:38,462 WARN [Master:0;asf001.sp2.ygridcore.net,33796,1365336878036] hbase.ZNodeClearer(58): No filename given to save the znode used, it won't be saved (Environment variable HBASE_ZNODE_FILE is not set). 2013-04-07 12:14:38,462 INFO [Master:0;asf001.sp2.ygridcore.net,33796,1365336878036] master.ActiveMasterManager(159): Master=asf001.sp2.ygridcore.net,33796,1365336878036 2013-04-07 12:14:38,472 INFO [Master:0;asf001.sp2.ygridcore.net,33796,1365336878036] master.SplitLogManager(181): timeout = 120000 2013-04-07 12:14:38,472 INFO [Master:0;asf001.sp2.ygridcore.net,33796,1365336878036] master.SplitLogManager(182): unassigned timeout = 180000 2013-04-07 12:14:38,477 INFO [Master:0;asf001.sp2.ygridcore.net,33796,1365336878036] master.SplitLogManager(858): found 0 orphan tasks and 0 rescan nodes 2013-04-07 12:14:38,533 DEBUG [Master:0;asf001.sp2.ygridcore.net,33796,1365336878036] util.FSUtils(640): Created cluster ID file at hdfs://localhost:60405/user/jenkins/hbase/hbase.id with ID: cd034c7a-08a3-4d5e-8eea-f9d6aa6dd983 2013-04-07 12:14:38,549 INFO [pool-1-thread-1] regionserver.ShutdownHook(87): Installed shutdown hook thread: Shutdownhook:RegionServer:0;asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:14:38,549 INFO [pool-1-thread-1] regionserver.ShutdownHook(87): Installed shutdown hook thread: Shutdownhook:RegionServer:1;asf001.sp2.ygridcore.net,35684,1365336878445 2013-04-07 12:14:38,553 DEBUG [RegionServer:0;asf001.sp2.ygridcore.net,60884,1365336878389] zookeeper.ZKConfig(84): Skipped reading ZK properties file 'zoo.cfg' since 'hbase.config.read.zookeeper.config' was not set to true 2013-04-07 12:14:38,554 DEBUG [RegionServer:0;asf001.sp2.ygridcore.net,60884,1365336878389] zookeeper.ZKUtil(119): regionserver:60884 opening connection to ZooKeeper with ensemble (localhost:51597) 2013-04-07 12:14:38,553 DEBUG [RegionServer:1;asf001.sp2.ygridcore.net,35684,1365336878445] zookeeper.ZKConfig(84): Skipped reading ZK properties file 'zoo.cfg' since 'hbase.config.read.zookeeper.config' was not set to true 2013-04-07 12:14:38,555 INFO [RegionServer:0;asf001.sp2.ygridcore.net,60884,1365336878389] zookeeper.RecoverableZooKeeper(119): The identifier of this process is regionserver:60884 2013-04-07 12:14:38,559 DEBUG [RegionServer:1;asf001.sp2.ygridcore.net,35684,1365336878445] zookeeper.ZKUtil(119): regionserver:35684 opening connection to ZooKeeper with ensemble (localhost:51597) 2013-04-07 12:14:38,561 INFO [RegionServer:1;asf001.sp2.ygridcore.net,35684,1365336878445] zookeeper.RecoverableZooKeeper(119): The identifier of this process is regionserver:35684 2013-04-07 12:14:38,563 DEBUG [RegionServer:0;asf001.sp2.ygridcore.net,60884,1365336878389-EventThread] zookeeper.ZooKeeperWatcher(274): regionserver:60884 Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2013-04-07 12:14:38,564 DEBUG [RegionServer:0;asf001.sp2.ygridcore.net,60884,1365336878389-EventThread] zookeeper.ZooKeeperWatcher(351): regionserver:60884-0x13de46b5f8b0001 connected 2013-04-07 12:14:38,565 DEBUG [RegionServer:0;asf001.sp2.ygridcore.net,60884,1365336878389] zookeeper.ZKUtil(422): regionserver:60884-0x13de46b5f8b0001 Set watcher on existing znode /hbase/master 2013-04-07 12:14:38,566 DEBUG [RegionServer:1;asf001.sp2.ygridcore.net,35684,1365336878445-EventThread] zookeeper.ZooKeeperWatcher(274): regionserver:35684 Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2013-04-07 12:14:38,566 DEBUG [RegionServer:1;asf001.sp2.ygridcore.net,35684,1365336878445] zookeeper.ZKUtil(422): regionserver:35684 Set watcher on existing znode /hbase/master 2013-04-07 12:14:38,568 DEBUG [RegionServer:1;asf001.sp2.ygridcore.net,35684,1365336878445-EventThread] zookeeper.ZooKeeperWatcher(351): regionserver:35684-0x13de46b5f8b0002 connected 2013-04-07 12:14:38,571 DEBUG [RegionServer:0;asf001.sp2.ygridcore.net,60884,1365336878389] zookeeper.ZKUtil(1680): regionserver:60884-0x13de46b5f8b0001 Retrieved 43 byte(s) of data from znode /hbase/master and set watcher; PBUF\x0A%\x0A\x18asf001.sp2.y... 2013-04-07 12:14:38,572 DEBUG [RegionServer:1;asf001.sp2.ygridcore.net,35684,1365336878445] zookeeper.ZKUtil(1680): regionserver:35684-0x13de46b5f8b0002 Retrieved 43 byte(s) of data from znode /hbase/master and set watcher; PBUF\x0A%\x0A\x18asf001.sp2.y... 2013-04-07 12:14:38,572 DEBUG [RegionServer:0;asf001.sp2.ygridcore.net,60884,1365336878389] zookeeper.ZKUtil(424): regionserver:60884-0x13de46b5f8b0001 /hbase/running does not exist. Watcher is set. 2013-04-07 12:14:38,573 INFO [Master:0;asf001.sp2.ygridcore.net,33796,1365336878036] master.MasterFileSystem(480): BOOTSTRAP: creating first META region 2013-04-07 12:14:38,574 DEBUG [RegionServer:1;asf001.sp2.ygridcore.net,35684,1365336878445] zookeeper.ZKUtil(424): regionserver:35684-0x13de46b5f8b0002 /hbase/running does not exist. Watcher is set. 2013-04-07 12:14:38,574 INFO [Master:0;asf001.sp2.ygridcore.net,33796,1365336878036] regionserver.HRegion(3862): creating HRegion .META. HTD == '.META.', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}, {NAME => 'info', DATA_BLOCK_ENCODING => 'NONE', BLOOMFILTER => 'NONE', REPLICATION_SCOPE => '0', COMPRESSION => 'NONE', VERSIONS => '10', TTL => '2147483647', MIN_VERSIONS => '0', KEEP_DELETED_CELLS => 'false', BLOCKSIZE => '8192', ENCODE_ON_DISK => 'true', IN_MEMORY => 'false', BLOCKCACHE => 'false'} RootDir = hdfs://localhost:60405/user/jenkins/hbase Table name == .META. 2013-04-07 12:14:38,583 DEBUG [Master:0;asf001.sp2.ygridcore.net,33796,1365336878036] util.FSUtils(211): Creating file=hdfs://localhost:60405/user/jenkins/hbase/.META./1028785192/.regioninfo with permission=rwxrwxrwx 2013-04-07 12:14:38,602 INFO [Master:0;asf001.sp2.ygridcore.net,33796,1365336878036] wal.FSHLog(354): HLog configuration: blocksize=64 MB, rollsize=60.8 MB, enabled=true, optionallogflushinternal=1000ms 2013-04-07 12:14:38,626 DEBUG [Master:0;asf001.sp2.ygridcore.net,33796,1365336878036] wal.SequenceFileLogWriter(189): using new createWriter -- HADOOP-6840 2013-04-07 12:14:38,627 DEBUG [Master:0;asf001.sp2.ygridcore.net,33796,1365336878036] wal.SequenceFileLogWriter(193): Path=hdfs://localhost:60405/user/jenkins/hbase/.META./1028785192/.logs/hlog.1365336878609, compression=false 2013-04-07 12:14:38,627 INFO [Master:0;asf001.sp2.ygridcore.net,33796,1365336878036] wal.FSHLog(552): Rolled log; new path=/user/jenkins/hbase/.META./1028785192/.logs/hlog.1365336878609 2013-04-07 12:14:38,627 INFO [Master:0;asf001.sp2.ygridcore.net,33796,1365336878036] wal.FSHLog(448): Using getNumCurrentReplicas--HDFS-826 2013-04-07 12:14:38,639 DEBUG [Master:0;asf001.sp2.ygridcore.net,33796,1365336878036] regionserver.HRegion(496): Instantiated .META.,,1.1028785192 2013-04-07 12:14:38,674 INFO [StoreOpenerThread-.META.,,1.1028785192-1] compactions.CompactionConfiguration(85): Compaction configuration size [134217728, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; delete expired; major period 86400000, major jitter 0.200000 2013-04-07 12:14:38,693 INFO [StoreOpenerThread-.META.,,1.1028785192-1] util.ChecksumType$2(68): Checksum using org.apache.hadoop.util.PureJavaCrc32 2013-04-07 12:14:38,707 INFO [Master:0;asf001.sp2.ygridcore.net,33796,1365336878036] regionserver.HRegion(586): Onlined .META.,,1.1028785192; next sequenceid=1 2013-04-07 12:14:38,708 DEBUG [Master:0;asf001.sp2.ygridcore.net,33796,1365336878036] regionserver.HRegion(889): Closing .META.,,1.1028785192: disabling compactions & flushes 2013-04-07 12:14:38,708 DEBUG [Master:0;asf001.sp2.ygridcore.net,33796,1365336878036] regionserver.HRegion(911): Updates disabled for region .META.,,1.1028785192 2013-04-07 12:14:38,710 INFO [StoreCloserThread-.META.,,1.1028785192-1] regionserver.HStore(622): Closed info 2013-04-07 12:14:38,710 INFO [Master:0;asf001.sp2.ygridcore.net,33796,1365336878036] regionserver.HRegion(968): Closed .META.,,1.1028785192 2013-04-07 12:14:38,710 INFO [Master:0;asf001.sp2.ygridcore.net,33796,1365336878036.logSyncer] wal.FSHLog$LogSyncer(1030): Master:0;asf001.sp2.ygridcore.net,33796,1365336878036.logSyncer exiting 2013-04-07 12:14:38,711 DEBUG [Master:0;asf001.sp2.ygridcore.net,33796,1365336878036] wal.FSHLog(836): closing hlog writer in hdfs://localhost:60405/user/jenkins/hbase/.META./1028785192/.logs 2013-04-07 12:14:38,726 DEBUG [Master:0;asf001.sp2.ygridcore.net,33796,1365336878036] wal.FSHLog(796): Moved 1 log files to /user/jenkins/hbase/.META./1028785192/.oldlogs 2013-04-07 12:14:38,763 DEBUG [Master:0;asf001.sp2.ygridcore.net,33796,1365336878036] fs.HFileSystem(199): Starting addLocationsOrderInterceptor with class class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2013-04-07 12:14:38,775 INFO [Master:0;asf001.sp2.ygridcore.net,33796,1365336878036] fs.HFileSystem(246): Added intercepting call to namenode#getBlockLocations 2013-04-07 12:14:38,790 DEBUG [Master:0;asf001.sp2.ygridcore.net,33796,1365336878036] zookeeper.ZKConfig(84): Skipped reading ZK properties file 'zoo.cfg' since 'hbase.config.read.zookeeper.config' was not set to true 2013-04-07 12:14:38,790 DEBUG [Master:0;asf001.sp2.ygridcore.net,33796,1365336878036] zookeeper.ZKUtil(119): hconnection-0x1f327e opening connection to ZooKeeper with ensemble (localhost:51597) 2013-04-07 12:14:38,791 INFO [Master:0;asf001.sp2.ygridcore.net,33796,1365336878036] zookeeper.RecoverableZooKeeper(119): The identifier of this process is hconnection-0x1f327e 2013-04-07 12:14:38,793 DEBUG [Master:0;asf001.sp2.ygridcore.net,33796,1365336878036-EventThread] zookeeper.ZooKeeperWatcher(274): hconnection-0x1f327e Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2013-04-07 12:14:38,795 DEBUG [Master:0;asf001.sp2.ygridcore.net,33796,1365336878036-EventThread] zookeeper.ZooKeeperWatcher(351): hconnection-0x1f327e-0x13de46b5f8b0003 connected 2013-04-07 12:14:38,795 DEBUG [Master:0;asf001.sp2.ygridcore.net,33796,1365336878036] zookeeper.ZKUtil(1680): hconnection-0x1f327e-0x13de46b5f8b0003 Retrieved 42 byte(s) of data from znode /hbase/hbaseid; data=PBUF\x0A$cd034c7a-08a3-4d5e-8... 2013-04-07 12:14:38,795 INFO [Master:0;asf001.sp2.ygridcore.net,33796,1365336878036] client.HConnectionManager$HConnectionImplementation(699): ClusterId is cd034c7a-08a3-4d5e-8eea-f9d6aa6dd983 2013-04-07 12:14:38,807 DEBUG [Master:0;asf001.sp2.ygridcore.net,33796,1365336878036] catalog.CatalogTracker(192): Starting catalog tracker org.apache.hadoop.hbase.catalog.CatalogTracker@9b86ea 2013-04-07 12:14:38,808 DEBUG [Master:0;asf001.sp2.ygridcore.net,33796,1365336878036] zookeeper.ZKUtil(424): master:33796-0x13de46b5f8b0000 /hbase/meta-region-server does not exist. Watcher is set. 2013-04-07 12:14:38,812 DEBUG [Master:0;asf001.sp2.ygridcore.net,33796,1365336878036] zookeeper.ZKUtil(424): master:33796-0x13de46b5f8b0000 /hbase/balancer does not exist. Watcher is set. 2013-04-07 12:14:38,869 DEBUG [RegionServer:1;asf001.sp2.ygridcore.net,35684,1365336878445-EventThread] zookeeper.ZooKeeperWatcher(274): regionserver:35684-0x13de46b5f8b0002 Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2013-04-07 12:14:38,869 DEBUG [RegionServer:0;asf001.sp2.ygridcore.net,60884,1365336878389-EventThread] zookeeper.ZooKeeperWatcher(274): regionserver:60884-0x13de46b5f8b0001 Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2013-04-07 12:14:38,870 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(274): master:33796-0x13de46b5f8b0000 Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2013-04-07 12:14:38,872 DEBUG [RegionServer:1;asf001.sp2.ygridcore.net,35684,1365336878445-EventThread] zookeeper.ZKUtil(1680): regionserver:35684-0x13de46b5f8b0002 Retrieved 34 byte(s) of data from znode /hbase/running and set watcher; PBUF\x0A\x1CSun Apr 07 12:14:... 2013-04-07 12:14:38,873 DEBUG [RegionServer:1;asf001.sp2.ygridcore.net,35684,1365336878445] zookeeper.ZKConfig(84): Skipped reading ZK properties file 'zoo.cfg' since 'hbase.config.read.zookeeper.config' was not set to true 2013-04-07 12:14:38,874 DEBUG [RegionServer:1;asf001.sp2.ygridcore.net,35684,1365336878445] zookeeper.ZKUtil(119): hconnection-0xbbaa16 opening connection to ZooKeeper with ensemble (localhost:51597) 2013-04-07 12:14:38,875 DEBUG [RegionServer:0;asf001.sp2.ygridcore.net,60884,1365336878389-EventThread] zookeeper.ZKUtil(1680): regionserver:60884-0x13de46b5f8b0001 Retrieved 34 byte(s) of data from znode /hbase/running and set watcher; PBUF\x0A\x1CSun Apr 07 12:14:... 2013-04-07 12:14:38,876 INFO [RegionServer:1;asf001.sp2.ygridcore.net,35684,1365336878445] zookeeper.RecoverableZooKeeper(119): The identifier of this process is hconnection-0xbbaa16 2013-04-07 12:14:38,876 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZKUtil(1680): master:33796-0x13de46b5f8b0000 Retrieved 34 byte(s) of data from znode /hbase/running and set watcher; PBUF\x0A\x1CSun Apr 07 12:14:... 2013-04-07 12:14:38,877 INFO [Master:0;asf001.sp2.ygridcore.net,33796,1365336878036] master.HMaster(615): Server active/primary master; asf001.sp2.ygridcore.net,33796,1365336878036, sessionid=0x13de46b5f8b0000, cluster-up flag was=false 2013-04-07 12:14:38,881 DEBUG [RegionServer:1;asf001.sp2.ygridcore.net,35684,1365336878445-EventThread] zookeeper.ZooKeeperWatcher(274): hconnection-0xbbaa16 Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2013-04-07 12:14:38,882 DEBUG [RegionServer:1;asf001.sp2.ygridcore.net,35684,1365336878445-EventThread] zookeeper.ZooKeeperWatcher(351): hconnection-0xbbaa16-0x13de46b5f8b0004 connected 2013-04-07 12:14:38,885 DEBUG [RegionServer:1;asf001.sp2.ygridcore.net,35684,1365336878445] zookeeper.ZKUtil(1680): hconnection-0xbbaa16-0x13de46b5f8b0004 Retrieved 42 byte(s) of data from znode /hbase/hbaseid; data=PBUF\x0A$cd034c7a-08a3-4d5e-8... 2013-04-07 12:14:38,885 INFO [RegionServer:1;asf001.sp2.ygridcore.net,35684,1365336878445] client.HConnectionManager$HConnectionImplementation(699): ClusterId is cd034c7a-08a3-4d5e-8eea-f9d6aa6dd983 2013-04-07 12:14:38,886 DEBUG [RegionServer:1;asf001.sp2.ygridcore.net,35684,1365336878445] catalog.CatalogTracker(192): Starting catalog tracker org.apache.hadoop.hbase.catalog.CatalogTracker@125d568 2013-04-07 12:14:38,887 DEBUG [RegionServer:0;asf001.sp2.ygridcore.net,60884,1365336878389] zookeeper.ZKConfig(84): Skipped reading ZK properties file 'zoo.cfg' since 'hbase.config.read.zookeeper.config' was not set to true 2013-04-07 12:14:38,888 DEBUG [RegionServer:0;asf001.sp2.ygridcore.net,60884,1365336878389] zookeeper.ZKUtil(119): hconnection-0x43ad4a opening connection to ZooKeeper with ensemble (localhost:51597) 2013-04-07 12:14:38,888 DEBUG [RegionServer:1;asf001.sp2.ygridcore.net,35684,1365336878445] zookeeper.ZKUtil(424): regionserver:35684-0x13de46b5f8b0002 /hbase/meta-region-server does not exist. Watcher is set. 2013-04-07 12:14:38,889 INFO [RegionServer:0;asf001.sp2.ygridcore.net,60884,1365336878389] zookeeper.RecoverableZooKeeper(119): The identifier of this process is hconnection-0x43ad4a 2013-04-07 12:14:38,892 DEBUG [RegionServer:1;asf001.sp2.ygridcore.net,35684,1365336878445] zookeeper.ZKUtil(1680): regionserver:35684-0x13de46b5f8b0002 Retrieved 42 byte(s) of data from znode /hbase/hbaseid; data=PBUF\x0A$cd034c7a-08a3-4d5e-8... 2013-04-07 12:14:38,892 INFO [RegionServer:1;asf001.sp2.ygridcore.net,35684,1365336878445] regionserver.HRegionServer(634): ClusterId : cd034c7a-08a3-4d5e-8eea-f9d6aa6dd983 2013-04-07 12:14:38,893 DEBUG [RegionServer:0;asf001.sp2.ygridcore.net,60884,1365336878389-EventThread] zookeeper.ZooKeeperWatcher(274): hconnection-0x43ad4a Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2013-04-07 12:14:38,895 DEBUG [RegionServer:0;asf001.sp2.ygridcore.net,60884,1365336878389-EventThread] zookeeper.ZooKeeperWatcher(351): hconnection-0x43ad4a-0x13de46b5f8b0005 connected 2013-04-07 12:14:38,896 DEBUG [RegionServer:0;asf001.sp2.ygridcore.net,60884,1365336878389] zookeeper.ZKUtil(1680): hconnection-0x43ad4a-0x13de46b5f8b0005 Retrieved 42 byte(s) of data from znode /hbase/hbaseid; data=PBUF\x0A$cd034c7a-08a3-4d5e-8... 2013-04-07 12:14:38,896 INFO [RegionServer:0;asf001.sp2.ygridcore.net,60884,1365336878389] client.HConnectionManager$HConnectionImplementation(699): ClusterId is cd034c7a-08a3-4d5e-8eea-f9d6aa6dd983 2013-04-07 12:14:38,896 DEBUG [RegionServer:0;asf001.sp2.ygridcore.net,60884,1365336878389] catalog.CatalogTracker(192): Starting catalog tracker org.apache.hadoop.hbase.catalog.CatalogTracker@17535b6 2013-04-07 12:14:38,900 DEBUG [RegionServer:0;asf001.sp2.ygridcore.net,60884,1365336878389] zookeeper.ZKUtil(424): regionserver:60884-0x13de46b5f8b0001 /hbase/meta-region-server does not exist. Watcher is set. 2013-04-07 12:14:38,905 DEBUG [RegionServer:0;asf001.sp2.ygridcore.net,60884,1365336878389] zookeeper.ZKUtil(1680): regionserver:60884-0x13de46b5f8b0001 Retrieved 42 byte(s) of data from znode /hbase/hbaseid; data=PBUF\x0A$cd034c7a-08a3-4d5e-8... 2013-04-07 12:14:38,905 INFO [RegionServer:1;asf001.sp2.ygridcore.net,35684,1365336878445] zookeeper.RecoverableZooKeeper(460): Node /hbase/online-snapshot already exists and this is not a retry 2013-04-07 12:14:38,905 INFO [RegionServer:0;asf001.sp2.ygridcore.net,60884,1365336878389] regionserver.HRegionServer(634): ClusterId : cd034c7a-08a3-4d5e-8eea-f9d6aa6dd983 2013-04-07 12:14:38,909 INFO [RegionServer:1;asf001.sp2.ygridcore.net,35684,1365336878445] zookeeper.RecoverableZooKeeper(460): Node /hbase/online-snapshot/acquired already exists and this is not a retry 2013-04-07 12:14:38,909 INFO [RegionServer:0;asf001.sp2.ygridcore.net,60884,1365336878389] zookeeper.RecoverableZooKeeper(460): Node /hbase/online-snapshot/acquired already exists and this is not a retry 2013-04-07 12:14:38,915 INFO [RegionServer:1;asf001.sp2.ygridcore.net,35684,1365336878445] zookeeper.RecoverableZooKeeper(460): Node /hbase/online-snapshot/reached already exists and this is not a retry 2013-04-07 12:14:38,918 INFO [Master:0;asf001.sp2.ygridcore.net,33796,1365336878036] procedure.ZKProcedureUtil(266): Clearing all procedure znodes: /hbase/online-snapshot/acquired /hbase/online-snapshot/reached /hbase/online-snapshot/abort 2013-04-07 12:14:38,919 INFO [RegionServer:0;asf001.sp2.ygridcore.net,60884,1365336878389] zookeeper.RecoverableZooKeeper(460): Node /hbase/online-snapshot/abort already exists and this is not a retry 2013-04-07 12:14:38,923 DEBUG [Master:0;asf001.sp2.ygridcore.net,33796,1365336878036] procedure.ZKProcedureCoordinatorRpcs(194): Starting the controller for procedure member:asf001.sp2.ygridcore.net,33796,1365336878036 2013-04-07 12:14:38,924 INFO [RegionServer:0;asf001.sp2.ygridcore.net,60884,1365336878389] regionserver.MemStoreFlusher(117): globalMemStoreLimit=675.6m, globalMemStoreLimitLowMark=591.1m, maxHeap=1.6g 2013-04-07 12:14:38,924 INFO [RegionServer:1;asf001.sp2.ygridcore.net,35684,1365336878445] regionserver.MemStoreFlusher(117): globalMemStoreLimit=675.6m, globalMemStoreLimitLowMark=591.1m, maxHeap=1.6g 2013-04-07 12:14:38,927 WARN [Master:0;asf001.sp2.ygridcore.net,33796,1365336878036] snapshot.SnapshotManager(242): Couldn't delete working snapshot directory: hdfs://localhost:60405/user/jenkins/hbase/.snapshot/.tmp 2013-04-07 12:14:38,928 INFO [RegionServer:0;asf001.sp2.ygridcore.net,60884,1365336878389] regionserver.HRegionServer$CompactionChecker(1237): Runs every 16mins, 40sec 2013-04-07 12:14:38,929 INFO [RegionServer:1;asf001.sp2.ygridcore.net,35684,1365336878445] regionserver.HRegionServer$CompactionChecker(1237): Runs every 16mins, 40sec 2013-04-07 12:14:38,934 DEBUG [Master:0;asf001.sp2.ygridcore.net,33796,1365336878036] executor.ExecutorService(99): Starting executor service name=MASTER_OPEN_REGION-asf001.sp2.ygridcore.net,33796,1365336878036, corePoolSize=5, maxPoolSize=5 2013-04-07 12:14:38,934 INFO [RegionServer:1;asf001.sp2.ygridcore.net,35684,1365336878445] regionserver.HRegionServer(1737): Attempting connect to Master server at asf001.sp2.ygridcore.net,33796,1365336878036 2013-04-07 12:14:38,934 DEBUG [Master:0;asf001.sp2.ygridcore.net,33796,1365336878036] executor.ExecutorService(99): Starting executor service name=MASTER_CLOSE_REGION-asf001.sp2.ygridcore.net,33796,1365336878036, corePoolSize=5, maxPoolSize=5 2013-04-07 12:14:38,934 INFO [RegionServer:0;asf001.sp2.ygridcore.net,60884,1365336878389] regionserver.HRegionServer(1737): Attempting connect to Master server at asf001.sp2.ygridcore.net,33796,1365336878036 2013-04-07 12:14:38,935 DEBUG [Master:0;asf001.sp2.ygridcore.net,33796,1365336878036] executor.ExecutorService(99): Starting executor service name=MASTER_SERVER_OPERATIONS-asf001.sp2.ygridcore.net,33796,1365336878036, corePoolSize=3, maxPoolSize=3 2013-04-07 12:14:38,935 DEBUG [Master:0;asf001.sp2.ygridcore.net,33796,1365336878036] executor.ExecutorService(99): Starting executor service name=MASTER_META_SERVER_OPERATIONS-asf001.sp2.ygridcore.net,33796,1365336878036, corePoolSize=5, maxPoolSize=5 2013-04-07 12:14:38,935 DEBUG [Master:0;asf001.sp2.ygridcore.net,33796,1365336878036] executor.ExecutorService(99): Starting executor service name=MASTER_TABLE_OPERATIONS-asf001.sp2.ygridcore.net,33796,1365336878036, corePoolSize=1, maxPoolSize=1 2013-04-07 12:14:38,937 DEBUG [Master:0;asf001.sp2.ygridcore.net,33796,1365336878036] cleaner.CleanerChore(86): initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveLogCleaner 2013-04-07 12:14:38,946 DEBUG [Master:0;asf001.sp2.ygridcore.net,33796,1365336878036] cleaner.CleanerChore(86): initialize cleaner=org.apache.hadoop.hbase.master.snapshot.SnapshotLogCleaner 2013-04-07 12:14:38,949 DEBUG [Master:0;asf001.sp2.ygridcore.net,33796,1365336878036] cleaner.CleanerChore(86): initialize cleaner=org.apache.hadoop.hbase.master.cleaner.HFileLinkCleaner 2013-04-07 12:14:38,950 DEBUG [Master:0;asf001.sp2.ygridcore.net,33796,1365336878036] cleaner.CleanerChore(86): initialize cleaner=org.apache.hadoop.hbase.master.snapshot.SnapshotHFileCleaner 2013-04-07 12:14:38,952 DEBUG [Master:0;asf001.sp2.ygridcore.net,33796,1365336878036] cleaner.CleanerChore(86): initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveHFileCleaner 2013-04-07 12:14:38,952 DEBUG [Master:0;asf001.sp2.ygridcore.net,33796,1365336878036] master.HMaster(1049): Started service threads 2013-04-07 12:14:38,952 INFO [Master:0;asf001.sp2.ygridcore.net,33796,1365336878036] master.ServerManager(778): Waiting for region servers count to settle; currently checked in 0, slept for 0 ms, expecting minimum of 2, maximum of 2, timeout of 4500 ms, interval of 1500 ms. 2013-04-07 12:14:38,955 INFO [RegionServer:0;asf001.sp2.ygridcore.net,60884,1365336878389] regionserver.HRegionServer(1744): Connected to master at asf001.sp2.ygridcore.net/67.195.138.31:33796 2013-04-07 12:14:38,955 INFO [RegionServer:1;asf001.sp2.ygridcore.net,35684,1365336878445] regionserver.HRegionServer(1744): Connected to master at asf001.sp2.ygridcore.net/67.195.138.31:33796 2013-04-07 12:14:38,955 INFO [RegionServer:0;asf001.sp2.ygridcore.net,60884,1365336878389] regionserver.HRegionServer(1790): Telling master at asf001.sp2.ygridcore.net,33796,1365336878036 that we are up with port=60884, startcode=1365336878389 2013-04-07 12:14:38,956 INFO [RegionServer:1;asf001.sp2.ygridcore.net,35684,1365336878445] regionserver.HRegionServer(1790): Telling master at asf001.sp2.ygridcore.net,33796,1365336878036 that we are up with port=35684, startcode=1365336878445 2013-04-07 12:14:38,990 INFO [IPC Server handler 0 on 33796] master.ServerManager(352): Registering server=asf001.sp2.ygridcore.net,35684,1365336878445 2013-04-07 12:14:38,990 INFO [IPC Server handler 1 on 33796] master.ServerManager(352): Registering server=asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:14:39,003 INFO [Master:0;asf001.sp2.ygridcore.net,33796,1365336878036] master.ServerManager(795): Finished waiting for region servers count to settle; checked in 2, slept for 50 ms, expecting minimum of 2, maximum of 2, master is running. 2013-04-07 12:14:39,006 DEBUG [RegionServer:0;asf001.sp2.ygridcore.net,60884,1365336878389] regionserver.HRegionServer(1084): Config from master: hbase.rootdir=hdfs://localhost:60405/user/jenkins/hbase 2013-04-07 12:14:39,006 DEBUG [RegionServer:1;asf001.sp2.ygridcore.net,35684,1365336878445] regionserver.HRegionServer(1084): Config from master: hbase.rootdir=hdfs://localhost:60405/user/jenkins/hbase 2013-04-07 12:14:39,008 DEBUG [RegionServer:0;asf001.sp2.ygridcore.net,60884,1365336878389] regionserver.HRegionServer(1084): Config from master: fs.default.name=hdfs://localhost:60405 2013-04-07 12:14:39,008 INFO [RegionServer:0;asf001.sp2.ygridcore.net,60884,1365336878389] regionserver.HRegionServer(1077): Master passed us hostname to use. Was=asf001.sp2.ygridcore.net, Now=asf001.sp2.ygridcore.net 2013-04-07 12:14:39,008 DEBUG [RegionServer:1;asf001.sp2.ygridcore.net,35684,1365336878445] regionserver.HRegionServer(1084): Config from master: fs.default.name=hdfs://localhost:60405 2013-04-07 12:14:39,009 INFO [RegionServer:1;asf001.sp2.ygridcore.net,35684,1365336878445] regionserver.HRegionServer(1077): Master passed us hostname to use. Was=asf001.sp2.ygridcore.net, Now=asf001.sp2.ygridcore.net 2013-04-07 12:14:39,011 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(274): master:33796-0x13de46b5f8b0000 Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2013-04-07 12:14:39,012 WARN [RegionServer:0;asf001.sp2.ygridcore.net,60884,1365336878389] hbase.ZNodeClearer(58): No filename given to save the znode used, it won't be saved (Environment variable HBASE_ZNODE_FILE is not set). 2013-04-07 12:14:39,012 WARN [RegionServer:1;asf001.sp2.ygridcore.net,35684,1365336878445] hbase.ZNodeClearer(58): No filename given to save the znode used, it won't be saved (Environment variable HBASE_ZNODE_FILE is not set). 2013-04-07 12:14:39,018 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZKUtil(422): master:33796-0x13de46b5f8b0000 Set watcher on existing znode /hbase/rs/asf001.sp2.ygridcore.net,35684,1365336878445 2013-04-07 12:14:39,020 DEBUG [RegionServer:0;asf001.sp2.ygridcore.net,60884,1365336878389] fs.HFileSystem(199): Starting addLocationsOrderInterceptor with class class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2013-04-07 12:14:39,020 INFO [RegionServer:0;asf001.sp2.ygridcore.net,60884,1365336878389] fs.HFileSystem(246): Added intercepting call to namenode#getBlockLocations 2013-04-07 12:14:39,021 DEBUG [RegionServer:0;asf001.sp2.ygridcore.net,60884,1365336878389] regionserver.HRegionServer(1300): logdir=hdfs://localhost:60405/user/jenkins/hbase/.logs/asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:14:39,022 DEBUG [RegionServer:1;asf001.sp2.ygridcore.net,35684,1365336878445] fs.HFileSystem(199): Starting addLocationsOrderInterceptor with class class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2013-04-07 12:14:39,022 INFO [RegionServer:1;asf001.sp2.ygridcore.net,35684,1365336878445] fs.HFileSystem(246): Added intercepting call to namenode#getBlockLocations 2013-04-07 12:14:39,022 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZKUtil(422): master:33796-0x13de46b5f8b0000 Set watcher on existing znode /hbase/rs/asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:14:39,024 DEBUG [RegionServer:1;asf001.sp2.ygridcore.net,35684,1365336878445] regionserver.HRegionServer(1300): logdir=hdfs://localhost:60405/user/jenkins/hbase/.logs/asf001.sp2.ygridcore.net,35684,1365336878445 2013-04-07 12:14:39,024 INFO [RegionServer:0;asf001.sp2.ygridcore.net,60884,1365336878389] wal.FSHLog(354): HLog configuration: blocksize=64 MB, rollsize=60.8 MB, enabled=true, optionallogflushinternal=1000ms 2013-04-07 12:14:39,026 INFO [RegionServer:1;asf001.sp2.ygridcore.net,35684,1365336878445] wal.FSHLog(354): HLog configuration: blocksize=64 MB, rollsize=60.8 MB, enabled=true, optionallogflushinternal=1000ms 2013-04-07 12:14:39,035 DEBUG [RegionServer:0;asf001.sp2.ygridcore.net,60884,1365336878389] wal.SequenceFileLogWriter(189): using new createWriter -- HADOOP-6840 2013-04-07 12:14:39,036 DEBUG [RegionServer:0;asf001.sp2.ygridcore.net,60884,1365336878389] wal.SequenceFileLogWriter(193): Path=hdfs://localhost:60405/user/jenkins/hbase/.logs/asf001.sp2.ygridcore.net,60884,1365336878389/asf001.sp2.ygridcore.net%2C60884%2C1365336878389.1365336879030, compression=false 2013-04-07 12:14:39,036 INFO [RegionServer:0;asf001.sp2.ygridcore.net,60884,1365336878389] wal.FSHLog(552): Rolled log; new path=/user/jenkins/hbase/.logs/asf001.sp2.ygridcore.net,60884,1365336878389/asf001.sp2.ygridcore.net%2C60884%2C1365336878389.1365336879030 2013-04-07 12:14:39,036 INFO [RegionServer:0;asf001.sp2.ygridcore.net,60884,1365336878389] wal.FSHLog(448): Using getNumCurrentReplicas--HDFS-826 2013-04-07 12:14:39,036 DEBUG [RegionServer:1;asf001.sp2.ygridcore.net,35684,1365336878445] wal.SequenceFileLogWriter(189): using new createWriter -- HADOOP-6840 2013-04-07 12:14:39,037 DEBUG [RegionServer:1;asf001.sp2.ygridcore.net,35684,1365336878445] wal.SequenceFileLogWriter(193): Path=hdfs://localhost:60405/user/jenkins/hbase/.logs/asf001.sp2.ygridcore.net,35684,1365336878445/asf001.sp2.ygridcore.net%2C35684%2C1365336878445.1365336879032, compression=false 2013-04-07 12:14:39,037 INFO [RegionServer:1;asf001.sp2.ygridcore.net,35684,1365336878445] wal.FSHLog(552): Rolled log; new path=/user/jenkins/hbase/.logs/asf001.sp2.ygridcore.net,35684,1365336878445/asf001.sp2.ygridcore.net%2C35684%2C1365336878445.1365336879032 2013-04-07 12:14:39,037 INFO [RegionServer:1;asf001.sp2.ygridcore.net,35684,1365336878445] wal.FSHLog(448): Using getNumCurrentReplicas--HDFS-826 2013-04-07 12:14:39,046 DEBUG [RegionServer:1;asf001.sp2.ygridcore.net,35684,1365336878445] executor.ExecutorService(99): Starting executor service name=RS_OPEN_REGION-asf001.sp2.ygridcore.net,35684,1365336878445, corePoolSize=3, maxPoolSize=3 2013-04-07 12:14:39,046 DEBUG [RegionServer:0;asf001.sp2.ygridcore.net,60884,1365336878389] executor.ExecutorService(99): Starting executor service name=RS_OPEN_REGION-asf001.sp2.ygridcore.net,60884,1365336878389, corePoolSize=3, maxPoolSize=3 2013-04-07 12:14:39,046 DEBUG [RegionServer:1;asf001.sp2.ygridcore.net,35684,1365336878445] executor.ExecutorService(99): Starting executor service name=RS_OPEN_META-asf001.sp2.ygridcore.net,35684,1365336878445, corePoolSize=1, maxPoolSize=1 2013-04-07 12:14:39,046 DEBUG [RegionServer:0;asf001.sp2.ygridcore.net,60884,1365336878389] executor.ExecutorService(99): Starting executor service name=RS_OPEN_META-asf001.sp2.ygridcore.net,60884,1365336878389, corePoolSize=1, maxPoolSize=1 2013-04-07 12:14:39,046 DEBUG [RegionServer:1;asf001.sp2.ygridcore.net,35684,1365336878445] executor.ExecutorService(99): Starting executor service name=RS_CLOSE_REGION-asf001.sp2.ygridcore.net,35684,1365336878445, corePoolSize=3, maxPoolSize=3 2013-04-07 12:14:39,046 DEBUG [RegionServer:0;asf001.sp2.ygridcore.net,60884,1365336878389] executor.ExecutorService(99): Starting executor service name=RS_CLOSE_REGION-asf001.sp2.ygridcore.net,60884,1365336878389, corePoolSize=3, maxPoolSize=3 2013-04-07 12:14:39,047 DEBUG [RegionServer:1;asf001.sp2.ygridcore.net,35684,1365336878445] executor.ExecutorService(99): Starting executor service name=RS_CLOSE_META-asf001.sp2.ygridcore.net,35684,1365336878445, corePoolSize=1, maxPoolSize=1 2013-04-07 12:14:39,047 DEBUG [RegionServer:0;asf001.sp2.ygridcore.net,60884,1365336878389] executor.ExecutorService(99): Starting executor service name=RS_CLOSE_META-asf001.sp2.ygridcore.net,60884,1365336878389, corePoolSize=1, maxPoolSize=1 2013-04-07 12:14:39,057 INFO [RegionServer:1;asf001.sp2.ygridcore.net,35684,1365336878445] regionserver.HRegionServer(1115): Serving as asf001.sp2.ygridcore.net,35684,1365336878445, RPC listening on asf001.sp2.ygridcore.net/67.195.138.31:35684, sessionid=0x13de46b5f8b0002 2013-04-07 12:14:39,057 INFO [SplitLogWorker-asf001.sp2.ygridcore.net,60884,1365336878389] regionserver.SplitLogWorker(135): SplitLogWorker asf001.sp2.ygridcore.net,60884,1365336878389 starting 2013-04-07 12:14:39,058 DEBUG [RegionServer:1;asf001.sp2.ygridcore.net,35684,1365336878445] procedure.ZKProcedureMemberRpcs(341): Starting procedure member 'asf001.sp2.ygridcore.net,35684,1365336878445' 2013-04-07 12:14:39,057 INFO [SplitLogWorker-asf001.sp2.ygridcore.net,35684,1365336878445] regionserver.SplitLogWorker(135): SplitLogWorker asf001.sp2.ygridcore.net,35684,1365336878445 starting 2013-04-07 12:14:39,057 INFO [RegionServer:0;asf001.sp2.ygridcore.net,60884,1365336878389] regionserver.HRegionServer(1115): Serving as asf001.sp2.ygridcore.net,60884,1365336878389, RPC listening on asf001.sp2.ygridcore.net/67.195.138.31:60884, sessionid=0x13de46b5f8b0001 2013-04-07 12:14:39,058 DEBUG [RegionServer:1;asf001.sp2.ygridcore.net,35684,1365336878445] procedure.ZKProcedureMemberRpcs(140): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2013-04-07 12:14:39,060 DEBUG [RegionServer:0;asf001.sp2.ygridcore.net,60884,1365336878389] procedure.ZKProcedureMemberRpcs(341): Starting procedure member 'asf001.sp2.ygridcore.net,60884,1365336878389' 2013-04-07 12:14:39,062 DEBUG [RegionServer:0;asf001.sp2.ygridcore.net,60884,1365336878389] procedure.ZKProcedureMemberRpcs(140): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2013-04-07 12:14:39,061 DEBUG [RegionServer:1;asf001.sp2.ygridcore.net,35684,1365336878445] procedure.ZKProcedureMemberRpcs(156): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2013-04-07 12:14:39,064 DEBUG [RegionServer:0;asf001.sp2.ygridcore.net,60884,1365336878389] procedure.ZKProcedureMemberRpcs(156): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2013-04-07 12:14:40,037 INFO [Master:0;asf001.sp2.ygridcore.net,33796,1365336878036] zookeeper.MetaRegionTracker(156): Unsetting META region location in ZooKeeper 2013-04-07 12:14:40,042 WARN [Master:0;asf001.sp2.ygridcore.net,33796,1365336878036] zookeeper.RecoverableZooKeeper(161): Node /hbase/meta-region-server already deleted, retry=false 2013-04-07 12:14:40,043 DEBUG [Master:0;asf001.sp2.ygridcore.net,33796,1365336878036] master.AssignmentManager(2039): No previous transition plan was found (or we are ignoring an existing plan) for .META.,,1.1028785192 so generated a random one; hri=.META.,,1.1028785192, src=, dest=asf001.sp2.ygridcore.net,35684,1365336878445; 2 (online=2, available=2) available servers, forceNewPlan=false 2013-04-07 12:14:40,044 INFO [Master:0;asf001.sp2.ygridcore.net,33796,1365336878036] master.RegionStates(264): Region {NAME => '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192,} transitioned from {.META.,,1.1028785192 state=OFFLINE, ts=1365336879007, server=null} to {.META.,,1.1028785192 state=OFFLINE, ts=1365336880043, server=null} 2013-04-07 12:14:40,044 DEBUG [Master:0;asf001.sp2.ygridcore.net,33796,1365336878036] zookeeper.ZKAssign(208): master:33796-0x13de46b5f8b0000 Creating (or updating) unassigned node for 1028785192 with OFFLINE state 2013-04-07 12:14:40,072 DEBUG [Master:0;asf001.sp2.ygridcore.net,33796,1365336878036] master.AssignmentManager(1761): Setting table .META. to ENABLED state. 2013-04-07 12:14:40,091 INFO [Master:0;asf001.sp2.ygridcore.net,33796,1365336878036] master.AssignmentManager(1780): Assigning region .META.,,1.1028785192 to asf001.sp2.ygridcore.net,35684,1365336878445 2013-04-07 12:14:40,091 INFO [Master:0;asf001.sp2.ygridcore.net,33796,1365336878036] master.RegionStates(264): Region {NAME => '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192,} transitioned from {.META.,,1.1028785192 state=OFFLINE, ts=1365336880043, server=null} to {.META.,,1.1028785192 state=PENDING_OPEN, ts=1365336880091, server=asf001.sp2.ygridcore.net,35684,1365336878445} 2013-04-07 12:14:40,091 DEBUG [Master:0;asf001.sp2.ygridcore.net,33796,1365336878036] master.ServerManager(713): New connection to asf001.sp2.ygridcore.net,35684,1365336878445 2013-04-07 12:14:40,119 INFO [PRI IPC Server handler 0 on 35684] regionserver.HRegionServer(3346): Received request to open region: .META.,,1.1028785192 on asf001.sp2.ygridcore.net,35684,1365336878445 2013-04-07 12:14:40,131 DEBUG [RS_OPEN_META-asf001.sp2.ygridcore.net,35684,1365336878445-0] zookeeper.ZKAssign(721): regionserver:35684-0x13de46b5f8b0002 Attempting to transition node 1028785192/.META. from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING 2013-04-07 12:14:40,131 INFO [Master:0;asf001.sp2.ygridcore.net,33796,1365336878036] master.ServerManager(533): AssignmentManager hasn't finished failover cleanup 2013-04-07 12:14:40,135 DEBUG [RS_OPEN_META-asf001.sp2.ygridcore.net,35684,1365336878445-0] zookeeper.ZKUtil(1680): regionserver:35684-0x13de46b5f8b0002 Retrieved 65 byte(s) of data from znode /hbase/region-in-transition/1028785192; data=region=.META.,,1, state=M_ZK_REGION_OFFLINE, servername=asf001.sp2.ygridcore.net,35684,1365336878445, createTime=1365336880047, payload.length=0 2013-04-07 12:14:40,138 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(274): master:33796-0x13de46b5f8b0000 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/region-in-transition/1028785192 2013-04-07 12:14:40,140 DEBUG [RS_OPEN_META-asf001.sp2.ygridcore.net,35684,1365336878445-0] zookeeper.ZKAssign(784): regionserver:35684-0x13de46b5f8b0002 Successfully transitioned node 1028785192 from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING 2013-04-07 12:14:40,140 DEBUG [RS_OPEN_META-asf001.sp2.ygridcore.net,35684,1365336878445-0] regionserver.HRegionServer(1319): logdir=hdfs://localhost:60405/user/jenkins/hbase/.logs/asf001.sp2.ygridcore.net,35684,1365336878445 2013-04-07 12:14:40,142 INFO [RS_OPEN_META-asf001.sp2.ygridcore.net,35684,1365336878445-0] wal.FSHLog(354): HLog configuration: blocksize=64 MB, rollsize=60.8 MB, enabled=true, optionallogflushinternal=1000ms 2013-04-07 12:14:40,142 DEBUG [hbase-am-zkevent-worker-pool-2-thread-1] zookeeper.ZKUtil(1680): master:33796-0x13de46b5f8b0000 Retrieved 63 byte(s) of data from znode /hbase/region-in-transition/1028785192 and set watcher; region=.META.,,1, state=RS_ZK_REGION_OPENING, servername=asf001.sp2.ygridcore.net,35684,1365336878445, createTime=1365336880135, payload.length=0 2013-04-07 12:14:40,143 DEBUG [hbase-am-zkevent-worker-pool-2-thread-1] master.AssignmentManager(740): Handling transition=RS_ZK_REGION_OPENING, server=asf001.sp2.ygridcore.net,35684,1365336878445, region=1028785192/.META., current state from region state map ={.META.,,1.1028785192 state=PENDING_OPEN, ts=1365336880091, server=asf001.sp2.ygridcore.net,35684,1365336878445} 2013-04-07 12:14:40,144 INFO [hbase-am-zkevent-worker-pool-2-thread-1] master.RegionStates(264): Region {NAME => '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192,} transitioned from {.META.,,1.1028785192 state=PENDING_OPEN, ts=1365336880091, server=asf001.sp2.ygridcore.net,35684,1365336878445} to {.META.,,1.1028785192 state=OPENING, ts=1365336880144, server=asf001.sp2.ygridcore.net,35684,1365336878445} 2013-04-07 12:14:40,152 DEBUG [RS_OPEN_META-asf001.sp2.ygridcore.net,35684,1365336878445-0] wal.SequenceFileLogWriter(189): using new createWriter -- HADOOP-6840 2013-04-07 12:14:40,152 DEBUG [RS_OPEN_META-asf001.sp2.ygridcore.net,35684,1365336878445-0] wal.SequenceFileLogWriter(193): Path=hdfs://localhost:60405/user/jenkins/hbase/.logs/asf001.sp2.ygridcore.net,35684,1365336878445/asf001.sp2.ygridcore.net%2C35684%2C1365336878445.1365336880146.meta, compression=false 2013-04-07 12:14:40,152 INFO [RS_OPEN_META-asf001.sp2.ygridcore.net,35684,1365336878445-0] wal.FSHLog(552): Rolled log; new path=/user/jenkins/hbase/.logs/asf001.sp2.ygridcore.net,35684,1365336878445/asf001.sp2.ygridcore.net%2C35684%2C1365336878445.1365336880146.meta 2013-04-07 12:14:40,152 INFO [RS_OPEN_META-asf001.sp2.ygridcore.net,35684,1365336878445-0] wal.FSHLog(448): Using getNumCurrentReplicas--HDFS-826 2013-04-07 12:14:40,154 INFO [RS_OPEN_META-asf001.sp2.ygridcore.net,35684,1365336878445-0] regionserver.HRegion(4023): HRegion.openHRegion Region name ==.META.,,1.1028785192 2013-04-07 12:14:40,154 DEBUG [RS_OPEN_META-asf001.sp2.ygridcore.net,35684,1365336878445-0] regionserver.HRegion(4025): Opening region: {NAME => '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192,} 2013-04-07 12:14:40,165 DEBUG [RS_OPEN_META-asf001.sp2.ygridcore.net,35684,1365336878445-0] coprocessor.CoprocessorHost(182): Loading coprocessor class org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint with path null and priority 536870911 2013-04-07 12:14:40,424 DEBUG [RS_OPEN_META-asf001.sp2.ygridcore.net,35684,1365336878445-0] regionserver.HRegion(5025): Registered coprocessor service: region=.META.,,1 service=MultiRowMutationService 2013-04-07 12:14:40,428 INFO [RS_OPEN_META-asf001.sp2.ygridcore.net,35684,1365336878445-0] regionserver.RegionCoprocessorHost(193): Load coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint from HTD of .META. successfully. 2013-04-07 12:14:40,431 DEBUG [RS_OPEN_META-asf001.sp2.ygridcore.net,35684,1365336878445-0] regionserver.MetricsRegionSourceImpl(54): Creating new MetricsRegionSourceImpl for table .META. 1028785192 2013-04-07 12:14:40,432 DEBUG [RS_OPEN_META-asf001.sp2.ygridcore.net,35684,1365336878445-0] regionserver.HRegion(496): Instantiated .META.,,1.1028785192 2013-04-07 12:14:40,439 INFO [StoreOpenerThread-.META.,,1.1028785192-1] compactions.CompactionConfiguration(85): Compaction configuration size [134217728, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; delete expired; major period 86400000, major jitter 0.200000 2013-04-07 12:14:40,446 INFO [RS_OPEN_META-asf001.sp2.ygridcore.net,35684,1365336878445-0] regionserver.HRegion(586): Onlined .META.,,1.1028785192; next sequenceid=1 2013-04-07 12:14:40,447 DEBUG [RS_OPEN_META-asf001.sp2.ygridcore.net,35684,1365336878445-0] zookeeper.ZKAssign(721): regionserver:35684-0x13de46b5f8b0002 Attempting to transition node 1028785192/.META. from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENING 2013-04-07 12:14:40,449 DEBUG [RS_OPEN_META-asf001.sp2.ygridcore.net,35684,1365336878445-0] zookeeper.ZKUtil(1680): regionserver:35684-0x13de46b5f8b0002 Retrieved 63 byte(s) of data from znode /hbase/region-in-transition/1028785192; data=region=.META.,,1, state=RS_ZK_REGION_OPENING, servername=asf001.sp2.ygridcore.net,35684,1365336878445, createTime=1365336880135, payload.length=0 2013-04-07 12:14:40,455 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(274): master:33796-0x13de46b5f8b0000 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/region-in-transition/1028785192 2013-04-07 12:14:40,455 DEBUG [RS_OPEN_META-asf001.sp2.ygridcore.net,35684,1365336878445-0] zookeeper.ZKAssign(784): regionserver:35684-0x13de46b5f8b0002 Successfully transitioned node 1028785192 from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENING 2013-04-07 12:14:40,456 INFO [PostOpenDeployTasks:1028785192] regionserver.HRegionServer(1563): Post open deploy tasks for region=.META.,,1.1028785192 2013-04-07 12:14:40,456 DEBUG [hbase-am-zkevent-worker-pool-2-thread-2] zookeeper.ZKUtil(1680): master:33796-0x13de46b5f8b0000 Retrieved 63 byte(s) of data from znode /hbase/region-in-transition/1028785192 and set watcher; region=.META.,,1, state=RS_ZK_REGION_OPENING, servername=asf001.sp2.ygridcore.net,35684,1365336878445, createTime=1365336880449, payload.length=0 2013-04-07 12:14:40,457 DEBUG [hbase-am-zkevent-worker-pool-2-thread-2] master.AssignmentManager(740): Handling transition=RS_ZK_REGION_OPENING, server=asf001.sp2.ygridcore.net,35684,1365336878445, region=1028785192/.META., current state from region state map ={.META.,,1.1028785192 state=OPENING, ts=1365336880144, server=asf001.sp2.ygridcore.net,35684,1365336878445} 2013-04-07 12:14:40,457 INFO [hbase-am-zkevent-worker-pool-2-thread-2] master.RegionStates(264): Region {NAME => '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192,} transitioned from {.META.,,1.1028785192 state=OPENING, ts=1365336880144, server=asf001.sp2.ygridcore.net,35684,1365336878445} to {.META.,,1.1028785192 state=OPENING, ts=1365336880457, server=asf001.sp2.ygridcore.net,35684,1365336878445} 2013-04-07 12:14:40,461 INFO [PostOpenDeployTasks:1028785192] zookeeper.MetaRegionTracker(122): Setting META region location in ZooKeeper as asf001.sp2.ygridcore.net,35684,1365336878445 2013-04-07 12:14:40,467 DEBUG [RegionServer:1;asf001.sp2.ygridcore.net,35684,1365336878445-EventThread] zookeeper.ZooKeeperWatcher(274): regionserver:35684-0x13de46b5f8b0002 Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/meta-region-server 2013-04-07 12:14:40,467 DEBUG [RegionServer:0;asf001.sp2.ygridcore.net,60884,1365336878389-EventThread] zookeeper.ZooKeeperWatcher(274): regionserver:60884-0x13de46b5f8b0001 Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/meta-region-server 2013-04-07 12:14:40,468 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(274): master:33796-0x13de46b5f8b0000 Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/meta-region-server 2013-04-07 12:14:40,472 DEBUG [RegionServer:0;asf001.sp2.ygridcore.net,60884,1365336878389-EventThread] zookeeper.ZKUtil(1680): regionserver:60884-0x13de46b5f8b0001 Retrieved 43 byte(s) of data from znode /hbase/meta-region-server and set watcher; asf001.sp2.ygridcore.net,35684,1365336878445 2013-04-07 12:14:40,472 DEBUG [RegionServer:1;asf001.sp2.ygridcore.net,35684,1365336878445-EventThread] zookeeper.ZKUtil(1680): regionserver:35684-0x13de46b5f8b0002 Retrieved 43 byte(s) of data from znode /hbase/meta-region-server and set watcher; asf001.sp2.ygridcore.net,35684,1365336878445 2013-04-07 12:14:40,473 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZKUtil(1680): master:33796-0x13de46b5f8b0000 Retrieved 43 byte(s) of data from znode /hbase/meta-region-server and set watcher; asf001.sp2.ygridcore.net,35684,1365336878445 2013-04-07 12:14:40,473 INFO [PostOpenDeployTasks:1028785192] regionserver.HRegionServer(1584): Done with post open deploy task for region=.META.,,1.1028785192 2013-04-07 12:14:40,473 DEBUG [RS_OPEN_META-asf001.sp2.ygridcore.net,35684,1365336878445-0] zookeeper.ZKAssign(721): regionserver:35684-0x13de46b5f8b0002 Attempting to transition node 1028785192/.META. from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENED 2013-04-07 12:14:40,475 DEBUG [RS_OPEN_META-asf001.sp2.ygridcore.net,35684,1365336878445-0] zookeeper.ZKUtil(1680): regionserver:35684-0x13de46b5f8b0002 Retrieved 63 byte(s) of data from znode /hbase/region-in-transition/1028785192; data=region=.META.,,1, state=RS_ZK_REGION_OPENING, servername=asf001.sp2.ygridcore.net,35684,1365336878445, createTime=1365336880449, payload.length=0 2013-04-07 12:14:40,479 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(274): master:33796-0x13de46b5f8b0000 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/region-in-transition/1028785192 2013-04-07 12:14:40,479 DEBUG [RS_OPEN_META-asf001.sp2.ygridcore.net,35684,1365336878445-0] zookeeper.ZKAssign(784): regionserver:35684-0x13de46b5f8b0002 Successfully transitioned node 1028785192 from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENED 2013-04-07 12:14:40,479 DEBUG [RS_OPEN_META-asf001.sp2.ygridcore.net,35684,1365336878445-0] handler.OpenRegionHandler(361): region transitioned to opened in zookeeper: {NAME => '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192,}, server: asf001.sp2.ygridcore.net,35684,1365336878445 2013-04-07 12:14:40,479 DEBUG [RS_OPEN_META-asf001.sp2.ygridcore.net,35684,1365336878445-0] handler.OpenRegionHandler(168): Opened .META.,,1.1028785192 on server:asf001.sp2.ygridcore.net,35684,1365336878445 2013-04-07 12:14:40,481 DEBUG [hbase-am-zkevent-worker-pool-2-thread-3] zookeeper.ZKUtil(1680): master:33796-0x13de46b5f8b0000 Retrieved 63 byte(s) of data from znode /hbase/region-in-transition/1028785192 and set watcher; region=.META.,,1, state=RS_ZK_REGION_OPENED, servername=asf001.sp2.ygridcore.net,35684,1365336878445, createTime=1365336880475, payload.length=0 2013-04-07 12:14:40,482 DEBUG [hbase-am-zkevent-worker-pool-2-thread-3] master.AssignmentManager(740): Handling transition=RS_ZK_REGION_OPENED, server=asf001.sp2.ygridcore.net,35684,1365336878445, region=1028785192/.META., current state from region state map ={.META.,,1.1028785192 state=OPENING, ts=1365336880457, server=asf001.sp2.ygridcore.net,35684,1365336878445} 2013-04-07 12:14:40,483 INFO [hbase-am-zkevent-worker-pool-2-thread-3] master.RegionStates(264): Region {NAME => '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192,} transitioned from {.META.,,1.1028785192 state=OPENING, ts=1365336880457, server=asf001.sp2.ygridcore.net,35684,1365336878445} to {.META.,,1.1028785192 state=OPEN, ts=1365336880483, server=asf001.sp2.ygridcore.net,35684,1365336878445} 2013-04-07 12:14:40,486 INFO [MASTER_OPEN_REGION-asf001.sp2.ygridcore.net,33796,1365336878036-0] handler.OpenedRegionHandler(144): Handling OPENED event for .META.,,1.1028785192 from asf001.sp2.ygridcore.net,35684,1365336878445; deleting unassigned node 2013-04-07 12:14:40,486 DEBUG [MASTER_OPEN_REGION-asf001.sp2.ygridcore.net,33796,1365336878036-0] zookeeper.ZKAssign(405): master:33796-0x13de46b5f8b0000 Deleting existing unassigned node for 1028785192 that is in expected state RS_ZK_REGION_OPENED 2013-04-07 12:14:40,489 DEBUG [MASTER_OPEN_REGION-asf001.sp2.ygridcore.net,33796,1365336878036-0] zookeeper.ZKUtil(1680): master:33796-0x13de46b5f8b0000 Retrieved 63 byte(s) of data from znode /hbase/region-in-transition/1028785192; data=region=.META.,,1, state=RS_ZK_REGION_OPENED, servername=asf001.sp2.ygridcore.net,35684,1365336878445, createTime=1365336880475, payload.length=0 2013-04-07 12:14:40,493 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(274): master:33796-0x13de46b5f8b0000 Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/region-in-transition/1028785192 2013-04-07 12:14:40,493 DEBUG [MASTER_OPEN_REGION-asf001.sp2.ygridcore.net,33796,1365336878036-0] zookeeper.ZKAssign(434): master:33796-0x13de46b5f8b0000 Successfully deleted unassigned node for region 1028785192 in expected state RS_ZK_REGION_OPENED 2013-04-07 12:14:40,495 DEBUG [hbase-am-zkevent-worker-pool-2-thread-4] master.AssignmentManager$4(1159): The znode of region .META.,,1.1028785192 has been deleted. 2013-04-07 12:14:40,495 INFO [hbase-am-zkevent-worker-pool-2-thread-4] master.RegionStates(264): Region {NAME => '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192,} transitioned from {.META.,,1.1028785192 state=OPEN, ts=1365336880483, server=asf001.sp2.ygridcore.net,35684,1365336878445} to {.META.,,1.1028785192 state=OPEN, ts=1365336880495, server=asf001.sp2.ygridcore.net,35684,1365336878445} 2013-04-07 12:14:40,495 INFO [hbase-am-zkevent-worker-pool-2-thread-4] master.RegionStates(299): Onlined region {NAME => '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192,} on asf001.sp2.ygridcore.net,35684,1365336878445 2013-04-07 12:14:40,495 INFO [hbase-am-zkevent-worker-pool-2-thread-4] master.AssignmentManager$4(1164): The master has opened the region .META.,,1.1028785192 that was online on asf001.sp2.ygridcore.net,35684,1365336878445 2013-04-07 12:14:40,498 DEBUG [Master:0;asf001.sp2.ygridcore.net,33796,1365336878036] zookeeper.ZKUtil(1680): master:33796-0x13de46b5f8b0000 Retrieved 43 byte(s) of data from znode /hbase/meta-region-server and set watcher; asf001.sp2.ygridcore.net,35684,1365336878445 2013-04-07 12:14:40,500 DEBUG [Master:0;asf001.sp2.ygridcore.net,33796,1365336878036] zookeeper.ZKUtil(1680): master:33796-0x13de46b5f8b0000 Retrieved 43 byte(s) of data from znode /hbase/meta-region-server and set watcher; asf001.sp2.ygridcore.net,35684,1365336878445 2013-04-07 12:14:40,500 INFO [Master:0;asf001.sp2.ygridcore.net,33796,1365336878036] master.HMaster(897): .META. assigned=1, rit=false, location=asf001.sp2.ygridcore.net,35684,1365336878445 2013-04-07 12:14:40,512 DEBUG [Master:0;asf001.sp2.ygridcore.net,33796,1365336878036] zookeeper.ZKUtil(1680): hconnection-0x1f327e-0x13de46b5f8b0003 Retrieved 43 byte(s) of data from znode /hbase/meta-region-server; data=asf001.sp2.ygridcore.net,35684,1365336878445 2013-04-07 12:14:40,514 DEBUG [Master:0;asf001.sp2.ygridcore.net,33796,1365336878036] client.ClientScanner(96): Creating scanner over .META. starting at key '' 2013-04-07 12:14:40,515 DEBUG [Master:0;asf001.sp2.ygridcore.net,33796,1365336878036] client.ClientScanner(209): Advancing internal scanner to startKey at '' 2013-04-07 12:14:40,517 DEBUG [Master:0;asf001.sp2.ygridcore.net,33796,1365336878036] zookeeper.ZKUtil(1680): hconnection-0x1f327e-0x13de46b5f8b0003 Retrieved 43 byte(s) of data from znode /hbase/meta-region-server; data=asf001.sp2.ygridcore.net,35684,1365336878445 2013-04-07 12:14:40,543 DEBUG [Master:0;asf001.sp2.ygridcore.net,33796,1365336878036] client.ClientScanner(196): Finished with scanning at {NAME => '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192,} 2013-04-07 12:14:40,543 INFO [Master:0;asf001.sp2.ygridcore.net,33796,1365336878036] catalog.MetaMigrationConvertingToPB(167): .META. is not migrated 2013-04-07 12:14:40,543 INFO [Master:0;asf001.sp2.ygridcore.net,33796,1365336878036] catalog.MetaMigrationConvertingToPB(135): META has Writable serializations, migrating META to PB serialization 2013-04-07 12:14:40,543 INFO [Master:0;asf001.sp2.ygridcore.net,33796,1365336878036] catalog.MetaMigrationConvertingToPB(152): Starting update of META 2013-04-07 12:14:40,545 DEBUG [Master:0;asf001.sp2.ygridcore.net,33796,1365336878036] zookeeper.ZKUtil(1680): hconnection-0x1f327e-0x13de46b5f8b0003 Retrieved 43 byte(s) of data from znode /hbase/meta-region-server; data=asf001.sp2.ygridcore.net,35684,1365336878445 2013-04-07 12:14:40,545 DEBUG [Master:0;asf001.sp2.ygridcore.net,33796,1365336878036] client.ClientScanner(96): Creating scanner over .META. starting at key '' 2013-04-07 12:14:40,546 DEBUG [Master:0;asf001.sp2.ygridcore.net,33796,1365336878036] client.ClientScanner(209): Advancing internal scanner to startKey at '' 2013-04-07 12:14:40,547 DEBUG [Master:0;asf001.sp2.ygridcore.net,33796,1365336878036] zookeeper.ZKUtil(1680): hconnection-0x1f327e-0x13de46b5f8b0003 Retrieved 43 byte(s) of data from znode /hbase/meta-region-server; data=asf001.sp2.ygridcore.net,35684,1365336878445 2013-04-07 12:14:40,550 DEBUG [Master:0;asf001.sp2.ygridcore.net,33796,1365336878036] client.ClientScanner(196): Finished with scanning at {NAME => '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192,} 2013-04-07 12:14:40,550 INFO [Master:0;asf001.sp2.ygridcore.net,33796,1365336878036] catalog.MetaMigrationConvertingToPB(155): Finished update of META. Total rows updated:0 2013-04-07 12:14:40,550 INFO [Master:0;asf001.sp2.ygridcore.net,33796,1365336878036] catalog.MetaMigrationConvertingToPB(138): ROOT and META updated with PB serialization. Total rows updated: 0 2013-04-07 12:14:40,553 DEBUG [Master:0;asf001.sp2.ygridcore.net,33796,1365336878036] zookeeper.ZKUtil(1680): master:33796-0x13de46b5f8b0000 Retrieved 6 byte(s) of data from znode /hbase/table/.META.; data=PBUF\x08\x00 2013-04-07 12:14:40,555 DEBUG [Master:0;asf001.sp2.ygridcore.net,33796,1365336878036] zookeeper.ZKUtil(1680): master:33796-0x13de46b5f8b0000 Retrieved 6 byte(s) of data from znode /hbase/table/.META.; data=PBUF\x08\x00 2013-04-07 12:14:40,557 DEBUG [Master:0;asf001.sp2.ygridcore.net,33796,1365336878036] zookeeper.ZKUtil(1680): master:33796-0x13de46b5f8b0000 Retrieved 6 byte(s) of data from znode /hbase/table/.META.; data=PBUF\x08\x00 2013-04-07 12:14:40,559 DEBUG [Master:0;asf001.sp2.ygridcore.net,33796,1365336878036] zookeeper.ZKUtil(1680): hconnection-0x1f327e-0x13de46b5f8b0003 Retrieved 43 byte(s) of data from znode /hbase/meta-region-server; data=asf001.sp2.ygridcore.net,35684,1365336878445 2013-04-07 12:14:40,559 DEBUG [Master:0;asf001.sp2.ygridcore.net,33796,1365336878036] client.ClientScanner(96): Creating scanner over .META. starting at key '' 2013-04-07 12:14:40,559 DEBUG [Master:0;asf001.sp2.ygridcore.net,33796,1365336878036] client.ClientScanner(209): Advancing internal scanner to startKey at '' 2013-04-07 12:14:40,560 DEBUG [Master:0;asf001.sp2.ygridcore.net,33796,1365336878036] zookeeper.ZKUtil(1680): hconnection-0x1f327e-0x13de46b5f8b0003 Retrieved 43 byte(s) of data from znode /hbase/meta-region-server; data=asf001.sp2.ygridcore.net,35684,1365336878445 2013-04-07 12:14:40,564 DEBUG [Master:0;asf001.sp2.ygridcore.net,33796,1365336878036] client.ClientScanner(196): Finished with scanning at {NAME => '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192,} 2013-04-07 12:14:40,565 INFO [Master:0;asf001.sp2.ygridcore.net,33796,1365336878036] master.AssignmentManager(445): Clean cluster startup. Assigning userregions 2013-04-07 12:14:40,565 DEBUG [Master:0;asf001.sp2.ygridcore.net,33796,1365336878036] zookeeper.ZKAssign(452): master:33796-0x13de46b5f8b0000 Deleting any existing unassigned nodes 2013-04-07 12:14:40,570 DEBUG [Master:0;asf001.sp2.ygridcore.net,33796,1365336878036] zookeeper.ZKUtil(1680): master:33796-0x13de46b5f8b0000 Retrieved 6 byte(s) of data from znode /hbase/table/.META.; data=PBUF\x08\x00 2013-04-07 12:14:40,572 DEBUG [Master:0;asf001.sp2.ygridcore.net,33796,1365336878036] zookeeper.ZKUtil(1680): master:33796-0x13de46b5f8b0000 Retrieved 6 byte(s) of data from znode /hbase/table/.META.; data=PBUF\x08\x00 2013-04-07 12:14:40,574 DEBUG [Master:0;asf001.sp2.ygridcore.net,33796,1365336878036] zookeeper.ZKUtil(1680): hconnection-0x1f327e-0x13de46b5f8b0003 Retrieved 43 byte(s) of data from znode /hbase/meta-region-server; data=asf001.sp2.ygridcore.net,35684,1365336878445 2013-04-07 12:14:40,575 DEBUG [Master:0;asf001.sp2.ygridcore.net,33796,1365336878036] client.ClientScanner(96): Creating scanner over .META. starting at key '' 2013-04-07 12:14:40,575 DEBUG [Master:0;asf001.sp2.ygridcore.net,33796,1365336878036] client.ClientScanner(209): Advancing internal scanner to startKey at '' 2013-04-07 12:14:40,576 DEBUG [Master:0;asf001.sp2.ygridcore.net,33796,1365336878036] zookeeper.ZKUtil(1680): hconnection-0x1f327e-0x13de46b5f8b0003 Retrieved 43 byte(s) of data from znode /hbase/meta-region-server; data=asf001.sp2.ygridcore.net,35684,1365336878445 2013-04-07 12:14:40,582 DEBUG [Master:0;asf001.sp2.ygridcore.net,33796,1365336878036] client.ClientScanner(196): Finished with scanning at {NAME => '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192,} 2013-04-07 12:14:40,585 DEBUG [Master:0;asf001.sp2.ygridcore.net,33796,1365336878036] zookeeper.ZKUtil(1680): master:33796-0x13de46b5f8b0000 Retrieved 6 byte(s) of data from znode /hbase/table/.META.; data=PBUF\x08\x00 2013-04-07 12:14:40,588 DEBUG [Master:0;asf001.sp2.ygridcore.net,33796,1365336878036] zookeeper.ZKUtil(1680): master:33796-0x13de46b5f8b0000 Retrieved 6 byte(s) of data from znode /hbase/table/.META.; data=PBUF\x08\x00 2013-04-07 12:14:40,599 INFO [Master:0;asf001.sp2.ygridcore.net,33796,1365336878036] master.HMaster(792): Master has completed initialization 2013-04-07 12:14:40,602 DEBUG [asf001.sp2.ygridcore.net,33796,1365336878036-CatalogJanitor] zookeeper.ZKUtil(1680): hconnection-0x1f327e-0x13de46b5f8b0003 Retrieved 43 byte(s) of data from znode /hbase/meta-region-server; data=asf001.sp2.ygridcore.net,35684,1365336878445 2013-04-07 12:14:40,603 DEBUG [asf001.sp2.ygridcore.net,33796,1365336878036-CatalogJanitor] client.ClientScanner(96): Creating scanner over .META. starting at key '' 2013-04-07 12:14:40,604 DEBUG [asf001.sp2.ygridcore.net,33796,1365336878036-CatalogJanitor] client.ClientScanner(209): Advancing internal scanner to startKey at '' 2013-04-07 12:14:40,605 DEBUG [asf001.sp2.ygridcore.net,33796,1365336878036-CatalogJanitor] zookeeper.ZKUtil(1680): hconnection-0x1f327e-0x13de46b5f8b0003 Retrieved 43 byte(s) of data from znode /hbase/meta-region-server; data=asf001.sp2.ygridcore.net,35684,1365336878445 2013-04-07 12:14:40,612 DEBUG [asf001.sp2.ygridcore.net,33796,1365336878036-CatalogJanitor] client.ClientScanner(196): Finished with scanning at {NAME => '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192,} 2013-04-07 12:14:40,614 DEBUG [asf001.sp2.ygridcore.net,33796,1365336878036-CatalogJanitor] master.CatalogJanitor(269): Scanned 0 catalog row(s), gc'd 0 unreferenced merged region(s) and 0 unreferenced parent region(s) 2013-04-07 12:14:40,676 DEBUG [pool-1-thread-1] zookeeper.ZKConfig(84): Skipped reading ZK properties file 'zoo.cfg' since 'hbase.config.read.zookeeper.config' was not set to true 2013-04-07 12:14:40,677 DEBUG [pool-1-thread-1] zookeeper.ZKUtil(119): hconnection-0x768dc8 opening connection to ZooKeeper with ensemble (localhost:51597) 2013-04-07 12:14:40,678 INFO [pool-1-thread-1] zookeeper.RecoverableZooKeeper(119): The identifier of this process is hconnection-0x768dc8 2013-04-07 12:14:40,682 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(274): hconnection-0x768dc8 Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2013-04-07 12:14:40,683 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(351): hconnection-0x768dc8-0x13de46b5f8b0006 connected 2013-04-07 12:14:40,684 DEBUG [pool-1-thread-1] zookeeper.ZKUtil(1680): hconnection-0x768dc8-0x13de46b5f8b0006 Retrieved 42 byte(s) of data from znode /hbase/hbaseid; data=PBUF\x0A$cd034c7a-08a3-4d5e-8... 2013-04-07 12:14:40,684 INFO [pool-1-thread-1] client.HConnectionManager$HConnectionImplementation(699): ClusterId is cd034c7a-08a3-4d5e-8eea-f9d6aa6dd983 2013-04-07 12:14:40,685 DEBUG [pool-1-thread-1] zookeeper.ZKUtil(1680): hconnection-0x768dc8-0x13de46b5f8b0006 Retrieved 43 byte(s) of data from znode /hbase/meta-region-server; data=asf001.sp2.ygridcore.net,35684,1365336878445 2013-04-07 12:14:40,685 DEBUG [pool-1-thread-1] client.ClientScanner(96): Creating scanner over .META. starting at key '' 2013-04-07 12:14:40,686 DEBUG [pool-1-thread-1] client.ClientScanner(209): Advancing internal scanner to startKey at '' 2013-04-07 12:14:40,687 DEBUG [pool-1-thread-1] zookeeper.ZKUtil(1680): hconnection-0x768dc8-0x13de46b5f8b0006 Retrieved 43 byte(s) of data from znode /hbase/meta-region-server; data=asf001.sp2.ygridcore.net,35684,1365336878445 2013-04-07 12:14:40,694 DEBUG [pool-1-thread-1] client.ClientScanner(196): Finished with scanning at {NAME => '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192,} 2013-04-07 12:14:40,695 INFO [pool-1-thread-1] client.HConnectionManager$HConnectionImplementation(1672): Closing zookeeper sessionid=0x13de46b5f8b0006 2013-04-07 12:14:40,869 DEBUG [pool-1-thread-1] zookeeper.ZKConfig(84): Skipped reading ZK properties file 'zoo.cfg' since 'hbase.config.read.zookeeper.config' was not set to true 2013-04-07 12:14:40,870 DEBUG [pool-1-thread-1] zookeeper.ZKUtil(119): hconnection-0xfe04c6 opening connection to ZooKeeper with ensemble (localhost:51597) 2013-04-07 12:14:40,871 INFO [pool-1-thread-1] zookeeper.RecoverableZooKeeper(119): The identifier of this process is hconnection-0xfe04c6 2013-04-07 12:14:40,915 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(274): hconnection-0xfe04c6 Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2013-04-07 12:14:40,917 DEBUG [pool-1-thread-1] zookeeper.ZKUtil(1680): hconnection-0xfe04c6 Retrieved 42 byte(s) of data from znode /hbase/hbaseid; data=PBUF\x0A$cd034c7a-08a3-4d5e-8... 2013-04-07 12:14:40,917 INFO [pool-1-thread-1] client.HConnectionManager$HConnectionImplementation(699): ClusterId is cd034c7a-08a3-4d5e-8eea-f9d6aa6dd983 2013-04-07 12:14:40,917 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(351): hconnection-0xfe04c6-0x13de46b5f8b0007 connected 2013-04-07 12:14:40,918 INFO [pool-1-thread-1] hbase.HBaseTestingUtility(846): Minicluster is up 2013-04-07 12:14:40,952 INFO [pool-1-thread-1] hbase.ResourceChecker(147): before: TestZooKeeper#testCreateSilentIsReallySilent Thread=203, OpenFileDescriptor=314, MaxFileDescriptor=60000, SystemLoadAverage=340, ProcessCount=175, AvailableMemoryMB=6904, ConnectionCount=4 2013-04-07 12:14:40,953 DEBUG [pool-1-thread-1] zookeeper.ZKConfig(84): Skipped reading ZK properties file 'zoo.cfg' since 'hbase.config.read.zookeeper.config' was not set to true 2013-04-07 12:14:40,960 DEBUG [pool-1-thread-1] zookeeper.ZKConfig(84): Skipped reading ZK properties file 'zoo.cfg' since 'hbase.config.read.zookeeper.config' was not set to true 2013-04-07 12:14:40,963 DEBUG [pool-1-thread-1] zookeeper.ZKUtil(119): testMasterAddressManagerFromZK opening connection to ZooKeeper with ensemble (localhost:51597) 2013-04-07 12:14:40,965 INFO [pool-1-thread-1] zookeeper.RecoverableZooKeeper(119): The identifier of this process is testMasterAddressManagerFromZK 2013-04-07 12:14:40,971 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(274): testMasterAddressManagerFromZK Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2013-04-07 12:14:40,973 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(351): testMasterAddressManagerFromZK-0x13de46b5f8b0009 connected 2013-04-07 12:14:41,065 INFO [pool-1-thread-1] hbase.ResourceChecker(171): after: TestZooKeeper#testCreateSilentIsReallySilent Thread=207 (was 203) - Thread LEAK? -, OpenFileDescriptor=329 (was 314) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=340 (was 340), ProcessCount=175 (was 175), AvailableMemoryMB=6904 (was 6904), ConnectionCount=4 (was 4) 2013-04-07 12:16:05,149 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(274): testMasterAddressManagerFromZK-0x13de46b5f8b0009 Received ZooKeeper Event, type=None, state=Disconnected, path=null 2013-04-07 12:16:05,149 DEBUG [Master:0;asf001.sp2.ygridcore.net,33796,1365336878036-EventThread] zookeeper.ZooKeeperWatcher(274): hconnection-0x1f327e-0x13de46b5f8b0003 Received ZooKeeper Event, type=None, state=Disconnected, path=null 2013-04-07 12:16:05,149 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(356): testMasterAddressManagerFromZK-0x13de46b5f8b0009 Received Disconnected from ZooKeeper, ignoring 2013-04-07 12:16:05,149 DEBUG [Master:0;asf001.sp2.ygridcore.net,33796,1365336878036-EventThread] zookeeper.ZooKeeperWatcher(356): hconnection-0x1f327e-0x13de46b5f8b0003 Received Disconnected from ZooKeeper, ignoring 2013-04-07 12:14:41,090 INFO [pool-1-thread-1] hbase.ResourceChecker(147): before: TestZooKeeper#testZNodeDeletes Thread=207, OpenFileDescriptor=331, MaxFileDescriptor=60000, SystemLoadAverage=340, ProcessCount=175, AvailableMemoryMB=6904, ConnectionCount=4 2013-04-07 12:14:41,091 DEBUG [pool-1-thread-1] zookeeper.ZKConfig(84): Skipped reading ZK properties file 'zoo.cfg' since 'hbase.config.read.zookeeper.config' was not set to true 2013-04-07 12:14:41,091 DEBUG [pool-1-thread-1] zookeeper.ZKUtil(119): org.apache.hadoop.hbase.TestZooKeeper opening connection to ZooKeeper with ensemble (localhost:51597) 2013-04-07 12:14:41,092 INFO [pool-1-thread-1] zookeeper.RecoverableZooKeeper(119): The identifier of this process is org.apache.hadoop.hbase.TestZooKeeper 2013-04-07 12:14:41,096 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(274): org.apache.hadoop.hbase.TestZooKeeper Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2013-04-07 12:14:41,097 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(351): org.apache.hadoop.hbase.TestZooKeeper-0x13de46b5f8b000b connected 2013-04-07 12:14:41,123 DEBUG [pool-1-thread-1] zookeeper.ZKUtil(1680): org.apache.hadoop.hbase.TestZooKeeper-0x13de46b5f8b000b Retrieved 0 byte(s) of data from znode /l1/l2/l3/l4; data=empty 2013-04-07 12:14:41,139 DEBUG [pool-1-thread-1] zookeeper.ZKUtil(775): org.apache.hadoop.hbase.TestZooKeeper-0x13de46b5f8b000b Unable to get data of znode /l1/l2/l3/l4 because node does not exist (not necessarily an error) 2013-04-07 12:14:41,143 DEBUG [pool-1-thread-1] zookeeper.ZKUtil(775): org.apache.hadoop.hbase.TestZooKeeper-0x13de46b5f8b000b Unable to get data of znode /l1/l2 because node does not exist (not necessarily an error) 2013-04-07 12:14:41,160 INFO [pool-1-thread-1] hbase.ResourceChecker(171): after: TestZooKeeper#testZNodeDeletes Thread=207 (was 207), OpenFileDescriptor=330 (was 331), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=340 (was 340), ProcessCount=175 (was 175), AvailableMemoryMB=6904 (was 6904), ConnectionCount=4 (was 4) 2013-04-07 12:16:05,149 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(274): org.apache.hadoop.hbase.TestZooKeeper-0x13de46b5f8b000b Received ZooKeeper Event, type=None, state=Disconnected, path=null 2013-04-07 12:16:05,149 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(356): org.apache.hadoop.hbase.TestZooKeeper-0x13de46b5f8b000b Received Disconnected from ZooKeeper, ignoring 2013-04-07 12:14:41,177 INFO [pool-1-thread-1] hbase.ResourceChecker(147): before: TestZooKeeper#testClusterKey Thread=207, OpenFileDescriptor=332, MaxFileDescriptor=60000, SystemLoadAverage=340, ProcessCount=175, AvailableMemoryMB=6903, ConnectionCount=4 2013-04-07 12:14:41,217 INFO [pool-1-thread-1] hbase.ResourceChecker(171): after: TestZooKeeper#testClusterKey Thread=207 (was 207), OpenFileDescriptor=334 (was 332) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=340 (was 340), ProcessCount=175 (was 175), AvailableMemoryMB=6903 (was 6903), ConnectionCount=4 (was 4) 2013-04-07 12:14:41,233 INFO [pool-1-thread-1] hbase.ResourceChecker(147): before: TestZooKeeper#testRegionServerSessionExpired Thread=207, OpenFileDescriptor=336, MaxFileDescriptor=60000, SystemLoadAverage=340, ProcessCount=175, AvailableMemoryMB=6903, ConnectionCount=4 2013-04-07 12:14:41,233 INFO [pool-1-thread-1] hbase.TestZooKeeper(200): Starting testRegionServerSessionExpired 2013-04-07 12:14:41,234 DEBUG [pool-1-thread-1] zookeeper.ZKConfig(84): Skipped reading ZK properties file 'zoo.cfg' since 'hbase.config.read.zookeeper.config' was not set to true 2013-04-07 12:14:41,238 INFO [pool-1-thread-1-EventThread] hbase.HBaseTestingUtility$1(1786): Monitor ZKW received event=WatchedEvent state:SyncConnected type:None path:null 2013-04-07 12:14:41,242 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(274): master:33796-0x13de46b5f8b0000 Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/asf001.sp2.ygridcore.net,35684,1365336878445 2013-04-07 12:14:41,242 INFO [pool-1-thread-1-EventThread] zookeeper.RegionServerTracker(94): RegionServer ephemeral node deleted, processing expiration [asf001.sp2.ygridcore.net,35684,1365336878445] 2013-04-07 12:14:41,242 INFO [pool-1-thread-1] hbase.HBaseTestingUtility(1802): ZK Closed Session 0x13de46b5f8b0002 2013-04-07 12:14:41,246 DEBUG [pool-1-thread-1-EventThread] master.AssignmentManager(2917): based on AM, current region=.META.,,1.1028785192 is on server=asf001.sp2.ygridcore.net,35684,1365336878445 server being checked: asf001.sp2.ygridcore.net,35684,1365336878445 2013-04-07 12:14:41,248 DEBUG [pool-1-thread-1-EventThread] master.ServerManager(495): Added=asf001.sp2.ygridcore.net,35684,1365336878445 to dead servers, submitted shutdown handler to be executed meta=true 2013-04-07 12:14:41,248 INFO [MASTER_META_SERVER_OPERATIONS-asf001.sp2.ygridcore.net,33796,1365336878036-0] handler.MetaServerShutdownHandler(52): Splitting META logs for asf001.sp2.ygridcore.net,35684,1365336878445 2013-04-07 12:14:41,248 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(274): master:33796-0x13de46b5f8b0000 Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2013-04-07 12:14:41,252 DEBUG [MASTER_META_SERVER_OPERATIONS-asf001.sp2.ygridcore.net,33796,1365336878036-0] master.MasterFileSystem(320): Renamed region directory: hdfs://localhost:60405/user/jenkins/hbase/.logs/asf001.sp2.ygridcore.net,35684,1365336878445-splitting 2013-04-07 12:14:41,252 INFO [MASTER_META_SERVER_OPERATIONS-asf001.sp2.ygridcore.net,33796,1365336878036-0] master.SplitLogManager(959): dead splitlog workers [asf001.sp2.ygridcore.net,35684,1365336878445] 2013-04-07 12:14:41,254 DEBUG [MASTER_META_SERVER_OPERATIONS-asf001.sp2.ygridcore.net,33796,1365336878036-0] master.SplitLogManager(268): Scheduling batch of logs to split 2013-04-07 12:14:41,255 INFO [MASTER_META_SERVER_OPERATIONS-asf001.sp2.ygridcore.net,33796,1365336878036-0] master.SplitLogManager(270): started splitting logs in [hdfs://localhost:60405/user/jenkins/hbase/.logs/asf001.sp2.ygridcore.net,35684,1365336878445-splitting] 2013-04-07 12:14:41,256 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZKUtil(422): master:33796-0x13de46b5f8b0000 Set watcher on existing znode /hbase/rs/asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:14:41,263 DEBUG [RegionServer:0;asf001.sp2.ygridcore.net,60884,1365336878389-EventThread] zookeeper.ZooKeeperWatcher(274): regionserver:60884-0x13de46b5f8b0001 Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/splitlog 2013-04-07 12:14:41,263 DEBUG [RegionServer:0;asf001.sp2.ygridcore.net,60884,1365336878389-EventThread] regionserver.SplitLogWorker(506): tasks arrived or departed 2013-04-07 12:14:41,264 DEBUG [pool-1-thread-1-EventThread] master.SplitLogManager(462): put up splitlog task at znode /hbase/splitlog/.logs%2Fasf001.sp2.ygridcore.net%2C35684%2C1365336878445-splitting%2Fasf001.sp2.ygridcore.net%252C35684%252C1365336878445.1365336880146.meta 2013-04-07 12:14:41,267 DEBUG [pool-1-thread-1-EventThread] master.SplitLogManager(504): task not yet acquired /hbase/splitlog/.logs%2Fasf001.sp2.ygridcore.net%2C35684%2C1365336878445-splitting%2Fasf001.sp2.ygridcore.net%252C35684%252C1365336878445.1365336880146.meta ver = 0 2013-04-07 12:14:41,267 DEBUG [SplitLogWorker-asf001.sp2.ygridcore.net,60884,1365336878389] zookeeper.ZKUtil(1680): regionserver:60884-0x13de46b5f8b0001 Retrieved 45 byte(s) of data from znode /hbase/splitlog/.logs%2Fasf001.sp2.ygridcore.net%2C35684%2C1365336878445-splitting%2Fasf001.sp2.ygridcore.net%252C35684%252C1365336878445.1365336880146.meta; data=PBUF\x08\x00\x12%\x0A\x18asf0... 2013-04-07 12:14:41,269 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(274): master:33796-0x13de46b5f8b0000 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/splitlog/.logs%2Fasf001.sp2.ygridcore.net%2C35684%2C1365336878445-splitting%2Fasf001.sp2.ygridcore.net%252C35684%252C1365336878445.1365336880146.meta 2013-04-07 12:14:41,269 INFO [SplitLogWorker-asf001.sp2.ygridcore.net,60884,1365336878389] regionserver.SplitLogWorker(267): worker asf001.sp2.ygridcore.net,60884,1365336878389 acquired task /hbase/splitlog/.logs%2Fasf001.sp2.ygridcore.net%2C35684%2C1365336878445-splitting%2Fasf001.sp2.ygridcore.net%252C35684%252C1365336878445.1365336880146.meta 2013-04-07 12:14:41,270 INFO [pool-1-thread-1-EventThread] master.SplitLogManager(577): task /hbase/splitlog/.logs%2Fasf001.sp2.ygridcore.net%2C35684%2C1365336878445-splitting%2Fasf001.sp2.ygridcore.net%252C35684%252C1365336878445.1365336880146.meta acquired by asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:14:41,277 INFO [SplitLogWorker-asf001.sp2.ygridcore.net,60884,1365336878389] wal.HLogSplitter(432): Splitting hlog: hdfs://localhost:60405/user/jenkins/hbase/.logs/asf001.sp2.ygridcore.net,35684,1365336878445-splitting/asf001.sp2.ygridcore.net%2C35684%2C1365336878445.1365336880146.meta, length=0 2013-04-07 12:14:41,277 WARN [SplitLogWorker-asf001.sp2.ygridcore.net,60884,1365336878389] wal.HLogSplitter(739): File hdfs://localhost:60405/user/jenkins/hbase/.logs/asf001.sp2.ygridcore.net,35684,1365336878445-splitting/asf001.sp2.ygridcore.net%2C35684%2C1365336878445.1365336880146.meta might be still open, length is 0 2013-04-07 12:14:41,278 INFO [SplitLogWorker-asf001.sp2.ygridcore.net,60884,1365336878389] util.FSHDFSUtils(56): Recovering file hdfs://localhost:60405/user/jenkins/hbase/.logs/asf001.sp2.ygridcore.net,35684,1365336878445-splitting/asf001.sp2.ygridcore.net%2C35684%2C1365336878445.1365336880146.meta 2013-04-07 12:14:41,279 WARN [IPC Server handler 8 on 60405] namenode.FSNamesystem(2321): BLOCK* internalReleaseLease: No blocks found, lease removed for /user/jenkins/hbase/.logs/asf001.sp2.ygridcore.net,35684,1365336878445-splitting/asf001.sp2.ygridcore.net%2C35684%2C1365336878445.1365336880146.meta 2013-04-07 12:14:41,280 INFO [SplitLogWorker-asf001.sp2.ygridcore.net,60884,1365336878389] util.FSHDFSUtils(72): Attempt 1 to recoverLease on file hdfs://localhost:60405/user/jenkins/hbase/.logs/asf001.sp2.ygridcore.net,35684,1365336878445-splitting/asf001.sp2.ygridcore.net%2C35684%2C1365336878445.1365336880146.meta returned false, trying for 2ms 2013-04-07 12:14:41,338 DEBUG [RegionServer:1;asf001.sp2.ygridcore.net,35684,1365336878445-EventThread] zookeeper.ZooKeeperWatcher(274): regionserver:35684-0x13de46b5f8b0002 Received ZooKeeper Event, type=None, state=Disconnected, path=null 2013-04-07 12:14:41,338 DEBUG [RegionServer:1;asf001.sp2.ygridcore.net,35684,1365336878445-EventThread] zookeeper.ZooKeeperWatcher(356): regionserver:35684-0x13de46b5f8b0002 Received Disconnected from ZooKeeper, ignoring 2013-04-07 12:14:41,339 INFO [pool-1-thread-1-EventThread] hbase.HBaseTestingUtility$1(1786): Monitor ZKW received event=WatchedEvent state:Disconnected type:None path:null 2013-04-07 12:14:41,341 INFO [pool-1-thread-1] hbase.TestZooKeeper(241): Creating table test1365336881340 2013-04-07 12:14:41,345 DEBUG [pool-1-thread-1] zookeeper.ZKUtil(1680): hconnection-0xfe04c6-0x13de46b5f8b0007 Retrieved 43 byte(s) of data from znode /hbase/master; data=PBUF\x0A%\x0A\x18asf001.sp2.y... 2013-04-07 12:14:41,436 DEBUG [IPC Server handler 4 on 33796] zookeeper.ZKUtil(1680): master:33796-0x13de46b5f8b0000 Retrieved 43 byte(s) of data from znode /hbase/meta-region-server and set watcher; asf001.sp2.ygridcore.net,35684,1365336878445 2013-04-07 12:14:41,436 DEBUG [IPC Server handler 4 on 33796] master.TableLockManager$ZKTableLockManager$TableLockImpl(225): Attempt to acquire table write lock on :test1365336881340 for:C_M_CREATE_TABLE 2013-04-07 12:14:41,456 DEBUG [IPC Server handler 4 on 33796] zookeeper.ZKUtil(1680): master:33796-0x13de46b5f8b0000 Retrieved 84 byte(s) of data from znode /hbase/table-lock/test1365336881340/write-master:337960000000000; data=PBUF\x0A\x11test1365336881340... 2013-04-07 12:14:41,457 DEBUG [IPC Server handler 4 on 33796] lock.ZKInterProcessLockBase(220): Successfully acquired a lock for /hbase/table-lock/test1365336881340/write-master:337960000000000 2013-04-07 12:14:41,457 DEBUG [IPC Server handler 4 on 33796] master.TableLockManager$ZKTableLockManager$TableLockImpl(246): Acquired table write lock on :test1365336881340 for:C_M_CREATE_TABLE 2013-04-07 12:14:41,459 DEBUG [IPC Server handler 4 on 33796] zookeeper.ZKUtil(1680): hconnection-0x1f327e-0x13de46b5f8b0003 Retrieved 43 byte(s) of data from znode /hbase/meta-region-server; data=asf001.sp2.ygridcore.net,35684,1365336878445 2013-04-07 12:14:41,460 DEBUG [IPC Server handler 4 on 33796] client.ClientScanner(96): Creating scanner over .META. starting at key 'test1365336881340,,' 2013-04-07 12:14:41,460 DEBUG [IPC Server handler 4 on 33796] client.ClientScanner(209): Advancing internal scanner to startKey at 'test1365336881340,,' 2013-04-07 12:14:41,461 DEBUG [IPC Server handler 4 on 33796] zookeeper.ZKUtil(1680): hconnection-0x1f327e-0x13de46b5f8b0003 Retrieved 43 byte(s) of data from znode /hbase/meta-region-server; data=asf001.sp2.ygridcore.net,35684,1365336878445 2013-04-07 12:14:41,467 DEBUG [IPC Server handler 4 on 33796] client.ClientScanner(196): Finished with scanning at {NAME => '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192,} 2013-04-07 12:14:41,476 INFO [MASTER_TABLE_OPERATIONS-asf001.sp2.ygridcore.net,33796,1365336878036-0] handler.CreateTableHandler(146): Attempting to create the table test1365336881340 2013-04-07 12:14:41,480 DEBUG [asf001.sp2.ygridcore.net,33796,1365336878036.splitLogManagerTimeoutMonitor] master.SplitLogManager$TimeoutMonitor(1013): total tasks = 1 unassigned = 0 2013-04-07 12:14:41,484 DEBUG [pool-1-thread-1] client.MetaScanner(198): Scanning .META. starting at row=test1365336881340,,00000000000000 for max=2147483647 rows using hconnection-0xfe04c6 2013-04-07 12:14:41,486 DEBUG [pool-1-thread-1] zookeeper.ZKUtil(1680): hconnection-0xfe04c6-0x13de46b5f8b0007 Retrieved 43 byte(s) of data from znode /hbase/meta-region-server; data=asf001.sp2.ygridcore.net,35684,1365336878445 2013-04-07 12:14:41,781 INFO [SplitLogWorker-asf001.sp2.ygridcore.net,60884,1365336878389] util.FSHDFSUtils(72): Attempt 2 to recoverLease on file hdfs://localhost:60405/user/jenkins/hbase/.logs/asf001.sp2.ygridcore.net,35684,1365336878445-splitting/asf001.sp2.ygridcore.net%2C35684%2C1365336878445.1365336880146.meta returned true, trying for 503ms 2013-04-07 12:14:41,788 WARN [SplitLogWorker-asf001.sp2.ygridcore.net,60884,1365336878389] wal.HLogSplitter(753): Could not open hdfs://localhost:60405/user/jenkins/hbase/.logs/asf001.sp2.ygridcore.net,35684,1365336878445-splitting/asf001.sp2.ygridcore.net%2C35684%2C1365336878445.1365336880146.meta for reading. File is empty java.io.EOFException at java.io.DataInputStream.readFully(DataInputStream.java:180) at java.io.DataInputStream.readFully(DataInputStream.java:152) at org.apache.hadoop.io.SequenceFile$Reader.init(SequenceFile.java:1512) at org.apache.hadoop.io.SequenceFile$Reader.(SequenceFile.java:1490) at org.apache.hadoop.io.SequenceFile$Reader.(SequenceFile.java:1479) at org.apache.hadoop.io.SequenceFile$Reader.(SequenceFile.java:1474) at org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogReader$WALReader.(SequenceFileLogReader.java:56) at org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogReader.init(SequenceFileLogReader.java:178) at org.apache.hadoop.hbase.regionserver.wal.HLogFactory.createReader(HLogFactory.java:91) at org.apache.hadoop.hbase.regionserver.wal.HLogSplitter.getReader(HLogSplitter.java:832) at org.apache.hadoop.hbase.regionserver.wal.HLogSplitter.getReader(HLogSplitter.java:745) at org.apache.hadoop.hbase.regionserver.wal.HLogSplitter.splitLogFile(HLogSplitter.java:436) at org.apache.hadoop.hbase.regionserver.wal.HLogSplitter.splitLogFile(HLogSplitter.java:397) at org.apache.hadoop.hbase.regionserver.SplitLogWorker$1.exec(SplitLogWorker.java:111) at org.apache.hadoop.hbase.regionserver.SplitLogWorker.grabTask(SplitLogWorker.java:274) at org.apache.hadoop.hbase.regionserver.SplitLogWorker.taskLoop(SplitLogWorker.java:195) at org.apache.hadoop.hbase.regionserver.SplitLogWorker.run(SplitLogWorker.java:162) at java.lang.Thread.run(Thread.java:662) 2013-04-07 12:14:41,790 WARN [SplitLogWorker-asf001.sp2.ygridcore.net,60884,1365336878389] wal.HLogSplitter(444): Nothing to split in log file hdfs://localhost:60405/user/jenkins/hbase/.logs/asf001.sp2.ygridcore.net,35684,1365336878445-splitting/asf001.sp2.ygridcore.net%2C35684%2C1365336878445.1365336880146.meta 2013-04-07 12:14:41,792 DEBUG [RegionServer:0;asf001.sp2.ygridcore.net,60884,1365336878389-EventThread] zookeeper.ZooKeeperWatcher(274): regionserver:60884-0x13de46b5f8b0001 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/splitlog/.logs%2Fasf001.sp2.ygridcore.net%2C35684%2C1365336878445-splitting%2Fasf001.sp2.ygridcore.net%252C35684%252C1365336878445.1365336880146.meta 2013-04-07 12:14:41,793 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(274): master:33796-0x13de46b5f8b0000 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/splitlog/.logs%2Fasf001.sp2.ygridcore.net%2C35684%2C1365336878445-splitting%2Fasf001.sp2.ygridcore.net%252C35684%252C1365336878445.1365336880146.meta 2013-04-07 12:14:41,793 INFO [SplitLogWorker-asf001.sp2.ygridcore.net,60884,1365336878389] regionserver.SplitLogWorker(381): successfully transitioned task /hbase/splitlog/.logs%2Fasf001.sp2.ygridcore.net%2C35684%2C1365336878445-splitting%2Fasf001.sp2.ygridcore.net%252C35684%252C1365336878445.1365336880146.meta to final state DONE asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:14:41,793 INFO [SplitLogWorker-asf001.sp2.ygridcore.net,60884,1365336878389] regionserver.SplitLogWorker(315): worker asf001.sp2.ygridcore.net,60884,1365336878389 done with task /hbase/splitlog/.logs%2Fasf001.sp2.ygridcore.net%2C35684%2C1365336878445-splitting%2Fasf001.sp2.ygridcore.net%252C35684%252C1365336878445.1365336880146.meta in 523ms 2013-04-07 12:14:41,794 INFO [pool-1-thread-1-EventThread] master.SplitLogManager(512): task /hbase/splitlog/.logs%2Fasf001.sp2.ygridcore.net%2C35684%2C1365336878445-splitting%2Fasf001.sp2.ygridcore.net%252C35684%252C1365336878445.1365336880146.meta entered state: DONE asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:14:41,801 DEBUG [pool-1-thread-1-EventThread] wal.HLogSplitter(600): Archived processed log hdfs://localhost:60405/user/jenkins/hbase/.logs/asf001.sp2.ygridcore.net,35684,1365336878445-splitting/asf001.sp2.ygridcore.net%2C35684%2C1365336878445.1365336880146.meta to hdfs://localhost:60405/user/jenkins/hbase/.oldlogs/asf001.sp2.ygridcore.net%2C35684%2C1365336878445.1365336880146.meta 2013-04-07 12:14:41,803 INFO [pool-1-thread-1-EventThread] master.SplitLogManager(425): Done splitting /hbase/splitlog/.logs%2Fasf001.sp2.ygridcore.net%2C35684%2C1365336878445-splitting%2Fasf001.sp2.ygridcore.net%252C35684%252C1365336878445.1365336880146.meta 2013-04-07 12:14:41,804 ERROR [IPC Server handler 1 on 60405] security.UserGroupInformation(1152): PriviledgedActionException as:jenkins cause:java.io.IOException: /user/jenkins/hbase/.logs/asf001.sp2.ygridcore.net,35684,1365336878445-splitting is non empty 2013-04-07 12:14:41,805 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(274): master:33796-0x13de46b5f8b0000 Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/splitlog/.logs%2Fasf001.sp2.ygridcore.net%2C35684%2C1365336878445-splitting%2Fasf001.sp2.ygridcore.net%252C35684%252C1365336878445.1365336880146.meta 2013-04-07 12:14:41,805 DEBUG [RegionServer:0;asf001.sp2.ygridcore.net,60884,1365336878389-EventThread] zookeeper.ZooKeeperWatcher(274): regionserver:60884-0x13de46b5f8b0001 Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/splitlog 2013-04-07 12:14:41,805 DEBUG [RegionServer:0;asf001.sp2.ygridcore.net,60884,1365336878389-EventThread] regionserver.SplitLogWorker(506): tasks arrived or departed 2013-04-07 12:14:41,805 DEBUG [pool-1-thread-1-EventThread] master.SplitLogManager$DeleteAsyncCallback(1191): deleted /hbase/splitlog/.logs%2Fasf001.sp2.ygridcore.net%2C35684%2C1365336878445-splitting%2Fasf001.sp2.ygridcore.net%252C35684%252C1365336878445.1365336880146.meta 2013-04-07 12:14:41,806 WARN [MASTER_META_SERVER_OPERATIONS-asf001.sp2.ygridcore.net,33796,1365336878036-0] master.SplitLogManager(306): returning success without actually splitting and deleting all the log files in path hdfs://localhost:60405/user/jenkins/hbase/.logs/asf001.sp2.ygridcore.net,35684,1365336878445-splitting 2013-04-07 12:14:41,806 INFO [MASTER_META_SERVER_OPERATIONS-asf001.sp2.ygridcore.net,33796,1365336878036-0] master.SplitLogManager(318): finished splitting (more than or equal to) 0 bytes in 1 log files in [hdfs://localhost:60405/user/jenkins/hbase/.logs/asf001.sp2.ygridcore.net,35684,1365336878445-splitting] in 551ms 2013-04-07 12:14:41,807 DEBUG [MASTER_META_SERVER_OPERATIONS-asf001.sp2.ygridcore.net,33796,1365336878036-0] master.AssignmentManager(2917): based on AM, current region=.META.,,1.1028785192 is on server=asf001.sp2.ygridcore.net,35684,1365336878445 server being checked: asf001.sp2.ygridcore.net,35684,1365336878445 2013-04-07 12:14:41,807 INFO [MASTER_META_SERVER_OPERATIONS-asf001.sp2.ygridcore.net,33796,1365336878036-0] handler.MetaServerShutdownHandler(67): Server asf001.sp2.ygridcore.net,35684,1365336878445 was carrying META. Trying to assign. 2013-04-07 12:14:41,807 DEBUG [MASTER_META_SERVER_OPERATIONS-asf001.sp2.ygridcore.net,33796,1365336878036-0] master.RegionStates(325): Offline a region with current state=OPEN, expected state=OFFLINE, assigned to server: asf001.sp2.ygridcore.net,35684,1365336878445, expected null 2013-04-07 12:14:41,807 INFO [MASTER_META_SERVER_OPERATIONS-asf001.sp2.ygridcore.net,33796,1365336878036-0] master.RegionStates(264): Region {NAME => '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192,} transitioned from {.META.,,1.1028785192 state=OPEN, ts=1365336880495, server=asf001.sp2.ygridcore.net,35684,1365336878445} to {.META.,,1.1028785192 state=OFFLINE, ts=1365336881807, server=null} 2013-04-07 12:14:41,807 INFO [MASTER_META_SERVER_OPERATIONS-asf001.sp2.ygridcore.net,33796,1365336878036-0] master.RegionStates(334): Offlined region {NAME => '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192,} from asf001.sp2.ygridcore.net,35684,1365336878445 2013-04-07 12:14:41,809 DEBUG [MASTER_META_SERVER_OPERATIONS-asf001.sp2.ygridcore.net,33796,1365336878036-0] zookeeper.ZKUtil(1680): master:33796-0x13de46b5f8b0000 Retrieved 43 byte(s) of data from znode /hbase/meta-region-server and set watcher; asf001.sp2.ygridcore.net,35684,1365336878445 2013-04-07 12:14:41,810 DEBUG [MASTER_META_SERVER_OPERATIONS-asf001.sp2.ygridcore.net,33796,1365336878036-0] zookeeper.ZKUtil(1680): master:33796-0x13de46b5f8b0000 Retrieved 43 byte(s) of data from znode /hbase/meta-region-server and set watcher; asf001.sp2.ygridcore.net,35684,1365336878445 2013-04-07 12:14:41,815 DEBUG [MASTER_META_SERVER_OPERATIONS-asf001.sp2.ygridcore.net,33796,1365336878036-0] zookeeper.ZKUtil(1680): master:33796-0x13de46b5f8b0000 Retrieved 43 byte(s) of data from znode /hbase/meta-region-server and set watcher; asf001.sp2.ygridcore.net,35684,1365336878445 2013-04-07 12:14:41,919 INFO [RegionOpenAndInitThread-test1365336881340-1] regionserver.HRegion(3862): creating HRegion test1365336881340 HTD == 'test1365336881340', {NAME => 'fam', DATA_BLOCK_ENCODING => 'NONE', BLOOMFILTER => 'NONE', REPLICATION_SCOPE => '0', VERSIONS => '3', COMPRESSION => 'NONE', MIN_VERSIONS => '0', TTL => '2147483647', KEEP_DELETED_CELLS => 'false', BLOCKSIZE => '65536', IN_MEMORY => 'false', ENCODE_ON_DISK => 'true', BLOCKCACHE => 'true'} RootDir = hdfs://localhost:60405/user/jenkins/hbase/.tmp Table name == test1365336881340 2013-04-07 12:14:41,924 DEBUG [RegionOpenAndInitThread-test1365336881340-1] util.FSUtils(211): Creating file=hdfs://localhost:60405/user/jenkins/hbase/.tmp/test1365336881340/c9e9e308a02801bb891879b6cf471381/.regioninfo with permission=rwxrwxrwx 2013-04-07 12:14:42,085 DEBUG [IPC Server handler 1 on 33796] master.ServerManager(332): Server REPORT rejected; currently processing asf001.sp2.ygridcore.net,35684,1365336878445 as dead server 2013-04-07 12:14:42,097 FATAL [RegionServer:1;asf001.sp2.ygridcore.net,35684,1365336878445] regionserver.HRegionServer(1607): ABORTING region server asf001.sp2.ygridcore.net,35684,1365336878445: Unhandled exception: org.apache.hadoop.hbase.exceptions.YouAreDeadException: Server REPORT rejected; currently processing asf001.sp2.ygridcore.net,35684,1365336878445 as dead server org.apache.hadoop.hbase.exceptions.YouAreDeadException: org.apache.hadoop.hbase.exceptions.YouAreDeadException: Server REPORT rejected; currently processing asf001.sp2.ygridcore.net,35684,1365336878445 as dead server at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27) at java.lang.reflect.Constructor.newInstance(Constructor.java:513) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:95) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:79) at org.apache.hadoop.hbase.ipc.ProtobufRpcClientEngine$Invoker.invoke(ProtobufRpcClientEngine.java:146) at $Proxy19.regionServerReport(Unknown Source) at org.apache.hadoop.hbase.regionserver.HRegionServer.tryRegionServerReport(HRegionServer.java:919) at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:769) at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.runRegionServer(MiniHBaseCluster.java:151) at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.access$000(MiniHBaseCluster.java:103) at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer$1.run(MiniHBaseCluster.java:135) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:337) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1131) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.hbase.util.Methods.call(Methods.java:41) at org.apache.hadoop.hbase.security.User.call(User.java:420) at org.apache.hadoop.hbase.security.User.access$300(User.java:51) at org.apache.hadoop.hbase.security.User$SecureHadoopUser.runAs(User.java:260) at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.run(MiniHBaseCluster.java:133) at java.lang.Thread.run(Thread.java:662) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException: org.apache.hadoop.hbase.exceptions.YouAreDeadException: Server REPORT rejected; currently processing asf001.sp2.ygridcore.net,35684,1365336878445 as dead server at org.apache.hadoop.hbase.master.ServerManager.checkIsDead(ServerManager.java:333) at org.apache.hadoop.hbase.master.ServerManager.regionServerReport(ServerManager.java:250) at org.apache.hadoop.hbase.master.HMaster.regionServerReport(HMaster.java:1179) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.hbase.ipc.ProtobufRpcServerEngine$Server.call(ProtobufRpcServerEngine.java:174) at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1871) at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:1321) at org.apache.hadoop.hbase.ipc.ProtobufRpcClientEngine$Invoker.invoke(ProtobufRpcClientEngine.java:131) ... 19 more 2013-04-07 12:14:42,098 FATAL [RegionServer:1;asf001.sp2.ygridcore.net,35684,1365336878445] regionserver.HRegionServer(1615): RegionServer abort: loaded coprocessors are: [org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint] 2013-04-07 12:14:42,100 ERROR [IPC Server handler 2 on 33796] master.HMaster(1198): Region server ??asf001.sp2.ygridcore.net,35684,1365336878445 reported a fatal error: ABORTING region server asf001.sp2.ygridcore.net,35684,1365336878445: Unhandled exception: org.apache.hadoop.hbase.exceptions.YouAreDeadException: Server REPORT rejected; currently processing asf001.sp2.ygridcore.net,35684,1365336878445 as dead server Cause: org.apache.hadoop.hbase.exceptions.YouAreDeadException: org.apache.hadoop.hbase.exceptions.YouAreDeadException: Server REPORT rejected; currently processing asf001.sp2.ygridcore.net,35684,1365336878445 as dead server at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27) at java.lang.reflect.Constructor.newInstance(Constructor.java:513) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:95) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:79) at org.apache.hadoop.hbase.ipc.ProtobufRpcClientEngine$Invoker.invoke(ProtobufRpcClientEngine.java:146) at $Proxy19.regionServerReport(Unknown Source) at org.apache.hadoop.hbase.regionserver.HRegionServer.tryRegionServerReport(HRegionServer.java:919) at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:769) at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.runRegionServer(MiniHBaseCluster.java:151) at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.access$000(MiniHBaseCluster.java:103) at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer$1.run(MiniHBaseCluster.java:135) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:337) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1131) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.hbase.util.Methods.call(Methods.java:41) at org.apache.hadoop.hbase.security.User.call(User.java:420) at org.apache.hadoop.hbase.security.User.access$300(User.java:51) at org.apache.hadoop.hbase.security.User$SecureHadoopUser.runAs(User.java:260) at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.run(MiniHBaseCluster.java:133) at java.lang.Thread.run(Thread.java:662) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException: org.apache.hadoop.hbase.exceptions.YouAreDeadException: Server REPORT rejected; currently processing asf001.sp2.ygridcore.net,35684,1365336878445 as dead server at org.apache.hadoop.hbase.master.ServerManager.checkIsDead(ServerManager.java:333) at org.apache.hadoop.hbase.master.ServerManager.regionServerReport(ServerManager.java:250) at org.apache.hadoop.hbase.master.HMaster.regionServerReport(HMaster.java:1179) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.hbase.ipc.ProtobufRpcServerEngine$Server.call(ProtobufRpcServerEngine.java:174) at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1871) at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:1321) at org.apache.hadoop.hbase.ipc.ProtobufRpcClientEngine$Invoker.invoke(ProtobufRpcClientEngine.java:131) ... 19 more 2013-04-07 12:14:42,102 INFO [RegionServer:1;asf001.sp2.ygridcore.net,35684,1365336878445] regionserver.HRegionServer(1545): STOPPED: Unhandled exception: org.apache.hadoop.hbase.exceptions.YouAreDeadException: Server REPORT rejected; currently processing asf001.sp2.ygridcore.net,35684,1365336878445 as dead server 2013-04-07 12:14:42,106 INFO [RegionServer:1;asf001.sp2.ygridcore.net,35684,1365336878445] regionserver.SplitLogWorker(519): Sending interrupt to stop the worker thread 2013-04-07 12:14:42,106 INFO [SplitLogWorker-asf001.sp2.ygridcore.net,35684,1365336878445] regionserver.SplitLogWorker(205): SplitLogWorker interrupted while waiting for task, exiting: java.lang.InterruptedException 2013-04-07 12:14:42,107 INFO [SplitLogWorker-asf001.sp2.ygridcore.net,35684,1365336878445] regionserver.SplitLogWorker(169): SplitLogWorker asf001.sp2.ygridcore.net,35684,1365336878445 exiting 2013-04-07 12:14:42,108 DEBUG [pool-1-thread-1.LruBlockCache.EvictionThread] hfile.LruBlockCache(448): Block cache LRU eviction started; Attempting to free -408721.95 KB of total=1.98 MB 2013-04-07 12:14:42,108 INFO [Thread-123] regionserver.MemStoreFlusher$FlushHandler(267): Thread-123 exiting 2013-04-07 12:14:42,108 INFO [RegionServer:1;asf001.sp2.ygridcore.net,35684,1365336878445.compactionChecker] hbase.Chore(94): RegionServer:1;asf001.sp2.ygridcore.net,35684,1365336878445.compactionChecker exiting 2013-04-07 12:14:42,108 INFO [RegionServer:1;asf001.sp2.ygridcore.net,35684,1365336878445] snapshot.RegionServerSnapshotManager(151): Stopping RegionServerSnapshotManager abruptly. 2013-04-07 12:14:42,108 INFO [RS_OPEN_META-asf001.sp2.ygridcore.net,35684,1365336878445-0MetaLogRoller] regionserver.LogRoller(119): LogRoller exiting. 2013-04-07 12:14:42,109 INFO [RegionServer:1;asf001.sp2.ygridcore.net,35684,1365336878445] regionserver.HRegionServer(829): aborting server asf001.sp2.ygridcore.net,35684,1365336878445 2013-04-07 12:14:42,108 INFO [RegionServer:1;asf001.sp2.ygridcore.net,35684,1365336878445.logRoller] regionserver.LogRoller(119): LogRoller exiting. 2013-04-07 12:14:42,109 DEBUG [RegionServer:1;asf001.sp2.ygridcore.net,35684,1365336878445] catalog.CatalogTracker(208): Stopping catalog tracker org.apache.hadoop.hbase.catalog.CatalogTracker@125d568 2013-04-07 12:14:42,109 INFO [RegionServer:1;asf001.sp2.ygridcore.net,35684,1365336878445] client.HConnectionManager$HConnectionImplementation(1672): Closing zookeeper sessionid=0x13de46b5f8b0004 2013-04-07 12:14:42,111 INFO [RegionServer:1;asf001.sp2.ygridcore.net,35684,1365336878445] snapshot.RegionServerSnapshotManager(151): Stopping RegionServerSnapshotManager abruptly. 2013-04-07 12:14:42,112 INFO [RegionServer:1;asf001.sp2.ygridcore.net,35684,1365336878445] regionserver.CompactSplitThread(318): Waiting for Split Thread to finish... 2013-04-07 12:14:42,112 INFO [RegionServer:1;asf001.sp2.ygridcore.net,35684,1365336878445] regionserver.CompactSplitThread(318): Waiting for Merge Thread to finish... 2013-04-07 12:14:42,112 INFO [RegionServer:1;asf001.sp2.ygridcore.net,35684,1365336878445] regionserver.CompactSplitThread(318): Waiting for Large Compaction Thread to finish... 2013-04-07 12:14:42,112 INFO [RegionServer:1;asf001.sp2.ygridcore.net,35684,1365336878445] regionserver.CompactSplitThread(318): Waiting for Small Compaction Thread to finish... 2013-04-07 12:14:42,113 INFO [RegionServer:1;asf001.sp2.ygridcore.net,35684,1365336878445] regionserver.HRegionServer(993): Waiting on 1 regions to close 2013-04-07 12:14:42,113 DEBUG [RS_CLOSE_META-asf001.sp2.ygridcore.net,35684,1365336878445-0] handler.CloseRegionHandler(125): Processing close of .META.,,1.1028785192 2013-04-07 12:14:42,113 DEBUG [RegionServer:1;asf001.sp2.ygridcore.net,35684,1365336878445] regionserver.HRegionServer(997): {1028785192=.META.,,1.1028785192} 2013-04-07 12:14:42,115 DEBUG [RS_CLOSE_META-asf001.sp2.ygridcore.net,35684,1365336878445-0] regionserver.HRegion(889): Closing .META.,,1.1028785192: disabling compactions & flushes 2013-04-07 12:14:42,115 DEBUG [RS_CLOSE_META-asf001.sp2.ygridcore.net,35684,1365336878445-0] regionserver.HRegion(911): Updates disabled for region .META.,,1.1028785192 2013-04-07 12:14:42,115 INFO [StoreCloserThread-.META.,,1.1028785192-1] regionserver.HStore(622): Closed info 2013-04-07 12:14:42,117 INFO [RS_CLOSE_META-asf001.sp2.ygridcore.net,35684,1365336878445-0] regionserver.HRegion(968): Closed .META.,,1.1028785192 2013-04-07 12:14:42,117 DEBUG [RS_CLOSE_META-asf001.sp2.ygridcore.net,35684,1365336878445-0] handler.CloseRegionHandler(177): Closed region .META.,,1.1028785192 2013-04-07 12:14:42,194 DEBUG [RegionServer:1;asf001.sp2.ygridcore.net,35684,1365336878445-EventThread] zookeeper.ZooKeeperWatcher(274): regionserver:35684-0x13de46b5f8b0002 Received ZooKeeper Event, type=None, state=Expired, path=null 2013-04-07 12:14:42,195 FATAL [RegionServer:1;asf001.sp2.ygridcore.net,35684,1365336878445-EventThread] regionserver.HRegionServer(1607): ABORTING region server asf001.sp2.ygridcore.net,35684,1365336878445: regionserver:35684-0x13de46b5f8b0002 regionserver:35684-0x13de46b5f8b0002 received expired from ZooKeeper, aborting org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.connectionEvent(ZooKeeperWatcher.java:364) at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.process(ZooKeeperWatcher.java:283) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:519) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495) 2013-04-07 12:14:42,195 FATAL [RegionServer:1;asf001.sp2.ygridcore.net,35684,1365336878445-EventThread] regionserver.HRegionServer(1615): RegionServer abort: loaded coprocessors are: [org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint] 2013-04-07 12:14:42,197 ERROR [IPC Server handler 3 on 33796] master.HMaster(1198): Region server ??asf001.sp2.ygridcore.net,35684,1365336878445 reported a fatal error: ABORTING region server asf001.sp2.ygridcore.net,35684,1365336878445: regionserver:35684-0x13de46b5f8b0002 regionserver:35684-0x13de46b5f8b0002 received expired from ZooKeeper, aborting Cause: org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.connectionEvent(ZooKeeperWatcher.java:364) at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.process(ZooKeeperWatcher.java:283) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:519) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495) 2013-04-07 12:14:42,198 INFO [RegionServer:1;asf001.sp2.ygridcore.net,35684,1365336878445-EventThread] regionserver.HRegionServer(1545): STOPPED: regionserver:35684-0x13de46b5f8b0002 regionserver:35684-0x13de46b5f8b0002 received expired from ZooKeeper, aborting 2013-04-07 12:14:42,315 INFO [RegionServer:1;asf001.sp2.ygridcore.net,35684,1365336878445] regionserver.HRegionServer(859): stopping server asf001.sp2.ygridcore.net,35684,1365336878445; all regions closed. 2013-04-07 12:14:42,316 INFO [RS_OPEN_META-asf001.sp2.ygridcore.net,35684,1365336878445-0.logSyncer] wal.FSHLog$LogSyncer(1030): RS_OPEN_META-asf001.sp2.ygridcore.net,35684,1365336878445-0.logSyncer exiting 2013-04-07 12:14:42,316 DEBUG [RegionServer:1;asf001.sp2.ygridcore.net,35684,1365336878445] wal.FSHLog(836): closing hlog writer in hdfs://localhost:60405/user/jenkins/hbase/.logs/asf001.sp2.ygridcore.net,35684,1365336878445 2013-04-07 12:14:42,317 ERROR [IPC Server handler 7 on 60405] security.UserGroupInformation(1152): PriviledgedActionException as:jenkins.hfs.1 cause:org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException: No lease on /user/jenkins/hbase/.logs/asf001.sp2.ygridcore.net,35684,1365336878445/asf001.sp2.ygridcore.net%2C35684%2C1365336878445.1365336880146.meta File does not exist. [Lease. Holder: DFSClient_hb_rs_asf001.sp2.ygridcore.net,35684,1365336878445_-2003770590_156, pendingcreates: 1] 2013-04-07 12:14:42,318 WARN [Thread-164] hdfs.DFSClient$DFSOutputStream$DataStreamer(3033): DataStreamer Exception: org.apache.hadoop.ipc.RemoteException: org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException: No lease on /user/jenkins/hbase/.logs/asf001.sp2.ygridcore.net,35684,1365336878445/asf001.sp2.ygridcore.net%2C35684%2C1365336878445.1365336880146.meta File does not exist. [Lease. Holder: DFSClient_hb_rs_asf001.sp2.ygridcore.net,35684,1365336878445_-2003770590_156, pendingcreates: 1] at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:1720) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:1711) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1619) at org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:736) at sun.reflect.GeneratedMethodAccessor24.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:578) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1393) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1389) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1149) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1387) at org.apache.hadoop.ipc.Client.call(Client.java:1107) at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:229) at $Proxy9.addBlock(Unknown Source) at sun.reflect.GeneratedMethodAccessor24.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:85) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:62) at $Proxy9.addBlock(Unknown Source) at sun.reflect.GeneratedMethodAccessor24.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:267) at $Proxy18.addBlock(Unknown Source) at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:3686) at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:3546) at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2600(DFSClient.java:2749) at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2989) 2013-04-07 12:14:42,318 WARN [Thread-164] hdfs.DFSClient$DFSOutputStream(3211): Error Recovery for block null bad datanode[0] nodes == null 2013-04-07 12:14:42,319 WARN [Thread-164] hdfs.DFSClient$DFSOutputStream(3235): Could not get block locations. Source file "/user/jenkins/hbase/.logs/asf001.sp2.ygridcore.net,35684,1365336878445/asf001.sp2.ygridcore.net%2C35684%2C1365336878445.1365336880146.meta" - Aborting... 2013-04-07 12:14:42,320 ERROR [RegionServer:1;asf001.sp2.ygridcore.net,35684,1365336878445] regionserver.HRegionServer(1034): Metalog close and delete failed org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException: org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException: No lease on /user/jenkins/hbase/.logs/asf001.sp2.ygridcore.net,35684,1365336878445/asf001.sp2.ygridcore.net%2C35684%2C1365336878445.1365336880146.meta File does not exist. [Lease. Holder: DFSClient_hb_rs_asf001.sp2.ygridcore.net,35684,1365336878445_-2003770590_156, pendingcreates: 1] at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:1720) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:1711) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1619) at org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:736) at sun.reflect.GeneratedMethodAccessor24.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:578) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1393) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1389) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1149) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1387) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27) at java.lang.reflect.Constructor.newInstance(Constructor.java:513) at org.apache.hadoop.hbase.RemoteExceptionHandler.decodeRemoteException(RemoteExceptionHandler.java:97) at org.apache.hadoop.hbase.RemoteExceptionHandler.checkThrowable(RemoteExceptionHandler.java:49) at org.apache.hadoop.hbase.regionserver.HRegionServer.closeWAL(HRegionServer.java:1034) at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:865) at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.runRegionServer(MiniHBaseCluster.java:151) at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.access$000(MiniHBaseCluster.java:103) at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer$1.run(MiniHBaseCluster.java:135) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:337) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1131) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.hbase.util.Methods.call(Methods.java:41) at org.apache.hadoop.hbase.security.User.call(User.java:420) at org.apache.hadoop.hbase.security.User.access$300(User.java:51) at org.apache.hadoop.hbase.security.User$SecureHadoopUser.runAs(User.java:260) at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.run(MiniHBaseCluster.java:133) at java.lang.Thread.run(Thread.java:662) 2013-04-07 12:14:42,320 INFO [RegionServer:1;asf001.sp2.ygridcore.net,35684,1365336878445.logSyncer] wal.FSHLog$LogSyncer(1030): RegionServer:1;asf001.sp2.ygridcore.net,35684,1365336878445.logSyncer exiting 2013-04-07 12:14:42,320 DEBUG [RegionServer:1;asf001.sp2.ygridcore.net,35684,1365336878445] wal.FSHLog(836): closing hlog writer in hdfs://localhost:60405/user/jenkins/hbase/.logs/asf001.sp2.ygridcore.net,35684,1365336878445 2013-04-07 12:14:42,321 ERROR [IPC Server handler 8 on 60405] security.UserGroupInformation(1152): PriviledgedActionException as:jenkins.hfs.1 cause:org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException: No lease on /user/jenkins/hbase/.logs/asf001.sp2.ygridcore.net,35684,1365336878445/asf001.sp2.ygridcore.net%2C35684%2C1365336878445.1365336879032 File does not exist. [Lease. Holder: DFSClient_hb_rs_asf001.sp2.ygridcore.net,35684,1365336878445_-2003770590_156, pendingcreates: 1] 2013-04-07 12:14:42,322 WARN [Thread-120] hdfs.DFSClient$DFSOutputStream$DataStreamer(3033): DataStreamer Exception: org.apache.hadoop.ipc.RemoteException: org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException: No lease on /user/jenkins/hbase/.logs/asf001.sp2.ygridcore.net,35684,1365336878445/asf001.sp2.ygridcore.net%2C35684%2C1365336878445.1365336879032 File does not exist. [Lease. Holder: DFSClient_hb_rs_asf001.sp2.ygridcore.net,35684,1365336878445_-2003770590_156, pendingcreates: 1] at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:1720) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:1711) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1619) at org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:736) at sun.reflect.GeneratedMethodAccessor24.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:578) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1393) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1389) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1149) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1387) at org.apache.hadoop.ipc.Client.call(Client.java:1107) at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:229) at $Proxy9.addBlock(Unknown Source) at sun.reflect.GeneratedMethodAccessor24.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:85) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:62) at $Proxy9.addBlock(Unknown Source) at sun.reflect.GeneratedMethodAccessor24.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:267) at $Proxy18.addBlock(Unknown Source) at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:3686) at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:3546) at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2600(DFSClient.java:2749) at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2989) 2013-04-07 12:14:42,322 WARN [Thread-120] hdfs.DFSClient$DFSOutputStream(3211): Error Recovery for block null bad datanode[0] nodes == null 2013-04-07 12:14:42,322 WARN [Thread-120] hdfs.DFSClient$DFSOutputStream(3235): Could not get block locations. Source file "/user/jenkins/hbase/.logs/asf001.sp2.ygridcore.net,35684,1365336878445/asf001.sp2.ygridcore.net%2C35684%2C1365336878445.1365336879032" - Aborting... 2013-04-07 12:14:42,323 ERROR [RegionServer:1;asf001.sp2.ygridcore.net,35684,1365336878445] regionserver.HRegionServer(1045): Close and delete failed org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException: org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException: No lease on /user/jenkins/hbase/.logs/asf001.sp2.ygridcore.net,35684,1365336878445/asf001.sp2.ygridcore.net%2C35684%2C1365336878445.1365336879032 File does not exist. [Lease. Holder: DFSClient_hb_rs_asf001.sp2.ygridcore.net,35684,1365336878445_-2003770590_156, pendingcreates: 1] at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:1720) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:1711) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1619) at org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:736) at sun.reflect.GeneratedMethodAccessor24.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:578) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1393) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1389) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1149) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1387) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27) at java.lang.reflect.Constructor.newInstance(Constructor.java:513) at org.apache.hadoop.hbase.RemoteExceptionHandler.decodeRemoteException(RemoteExceptionHandler.java:97) at org.apache.hadoop.hbase.RemoteExceptionHandler.checkThrowable(RemoteExceptionHandler.java:49) at org.apache.hadoop.hbase.regionserver.HRegionServer.closeWAL(HRegionServer.java:1045) at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:865) at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.runRegionServer(MiniHBaseCluster.java:151) at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.access$000(MiniHBaseCluster.java:103) at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer$1.run(MiniHBaseCluster.java:135) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:337) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1131) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.hbase.util.Methods.call(Methods.java:41) at org.apache.hadoop.hbase.security.User.call(User.java:420) at org.apache.hadoop.hbase.security.User.access$300(User.java:51) at org.apache.hadoop.hbase.security.User$SecureHadoopUser.runAs(User.java:260) at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.run(MiniHBaseCluster.java:133) at java.lang.Thread.run(Thread.java:662) 2013-04-07 12:14:42,361 DEBUG [RegionOpenAndInitThread-test1365336881340-1] regionserver.HRegion(496): Instantiated test1365336881340,,1365336881424.c9e9e308a02801bb891879b6cf471381. 2013-04-07 12:14:42,361 DEBUG [RegionOpenAndInitThread-test1365336881340-1] regionserver.HRegion(889): Closing test1365336881340,,1365336881424.c9e9e308a02801bb891879b6cf471381.: disabling compactions & flushes 2013-04-07 12:14:42,361 DEBUG [RegionOpenAndInitThread-test1365336881340-1] regionserver.HRegion(911): Updates disabled for region test1365336881340,,1365336881424.c9e9e308a02801bb891879b6cf471381. 2013-04-07 12:14:42,361 INFO [RegionOpenAndInitThread-test1365336881340-1] regionserver.HRegion(968): Closed test1365336881340,,1365336881424.c9e9e308a02801bb891879b6cf471381. 2013-04-07 12:14:42,368 DEBUG [MASTER_TABLE_OPERATIONS-asf001.sp2.ygridcore.net,33796,1365336878036-0] zookeeper.ZKUtil(1680): hconnection-0x1f327e-0x13de46b5f8b0003 Retrieved 43 byte(s) of data from znode /hbase/meta-region-server; data=asf001.sp2.ygridcore.net,35684,1365336878445 2013-04-07 12:14:42,373 DEBUG [MASTER_TABLE_OPERATIONS-asf001.sp2.ygridcore.net,33796,1365336878036-0] zookeeper.ZKUtil(1680): hconnection-0x1f327e-0x13de46b5f8b0003 Retrieved 43 byte(s) of data from znode /hbase/meta-region-server; data=asf001.sp2.ygridcore.net,35684,1365336878445 2013-04-07 12:14:42,384 INFO [MASTER_TABLE_OPERATIONS-asf001.sp2.ygridcore.net,33796,1365336878036-0] client.HConnectionManager$HConnectionImplementation$ServerErrorTracker(2571): Server tracker timeout is 2966084ms 2013-04-07 12:14:42,386 DEBUG [MASTER_TABLE_OPERATIONS-asf001.sp2.ygridcore.net,33796,1365336878036-0] zookeeper.ZKUtil(1680): hconnection-0x1f327e-0x13de46b5f8b0003 Retrieved 43 byte(s) of data from znode /hbase/meta-region-server; data=asf001.sp2.ygridcore.net,35684,1365336878445 2013-04-07 12:14:42,423 INFO [RegionServer:1;asf001.sp2.ygridcore.net,35684,1365336878445] regionserver.Leases(124): RegionServer:1;asf001.sp2.ygridcore.net,35684,1365336878445 closing leases 2013-04-07 12:14:42,424 INFO [RegionServer:1;asf001.sp2.ygridcore.net,35684,1365336878445] regionserver.Leases(131): RegionServer:1;asf001.sp2.ygridcore.net,35684,1365336878445 closed leases 2013-04-07 12:14:42,424 WARN [RegionServer:1;asf001.sp2.ygridcore.net,35684,1365336878445] zookeeper.RecoverableZooKeeper(236): Possibly transient ZooKeeper, quorum=localhost:51597, exception=org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /hbase/rs/asf001.sp2.ygridcore.net,35684,1365336878445 2013-04-07 12:14:42,424 INFO [RegionServer:1;asf001.sp2.ygridcore.net,35684,1365336878445] util.RetryCounter(54): Sleeping 2000ms before retry #1... 2013-04-07 12:14:42,494 DEBUG [pool-1-thread-1] client.MetaScanner(198): Scanning .META. starting at row=test1365336881340,,00000000000000 for max=2147483647 rows using hconnection-0xfe04c6 2013-04-07 12:14:42,496 DEBUG [pool-1-thread-1] zookeeper.ZKUtil(1680): hconnection-0xfe04c6-0x13de46b5f8b0007 Retrieved 43 byte(s) of data from znode /hbase/meta-region-server; data=asf001.sp2.ygridcore.net,35684,1365336878445 2013-04-07 12:14:42,497 WARN [pool-1-thread-1] client.ServerCallable(169): Received exception, tries=0, numRetries=10 message=Call to asf001.sp2.ygridcore.net/67.195.138.31:35684 failed on local exception: java.io.EOFException 2013-04-07 12:14:42,818 DEBUG [MASTER_META_SERVER_OPERATIONS-asf001.sp2.ygridcore.net,33796,1365336878036-0] zookeeper.ZKUtil(1680): master:33796-0x13de46b5f8b0000 Retrieved 43 byte(s) of data from znode /hbase/meta-region-server and set watcher; asf001.sp2.ygridcore.net,35684,1365336878445 2013-04-07 12:14:42,819 DEBUG [MASTER_META_SERVER_OPERATIONS-asf001.sp2.ygridcore.net,33796,1365336878036-0] zookeeper.ZKUtil(1680): master:33796-0x13de46b5f8b0000 Retrieved 43 byte(s) of data from znode /hbase/meta-region-server and set watcher; asf001.sp2.ygridcore.net,35684,1365336878445 2013-04-07 12:14:42,820 INFO [MASTER_META_SERVER_OPERATIONS-asf001.sp2.ygridcore.net,33796,1365336878036-0] catalog.CatalogTracker(399): Failed verification of .META.,,1 at address=asf001.sp2.ygridcore.net,35684,1365336878445, exception=java.io.EOFException 2013-04-07 12:14:42,820 INFO [MASTER_META_SERVER_OPERATIONS-asf001.sp2.ygridcore.net,33796,1365336878036-0] zookeeper.MetaRegionTracker(156): Unsetting META region location in ZooKeeper 2013-04-07 12:14:42,824 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(274): master:33796-0x13de46b5f8b0000 Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/meta-region-server 2013-04-07 12:14:42,824 DEBUG [RegionServer:0;asf001.sp2.ygridcore.net,60884,1365336878389-EventThread] zookeeper.ZooKeeperWatcher(274): regionserver:60884-0x13de46b5f8b0001 Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/meta-region-server 2013-04-07 12:14:42,825 DEBUG [MASTER_META_SERVER_OPERATIONS-asf001.sp2.ygridcore.net,33796,1365336878036-0] master.AssignmentManager(2039): No previous transition plan was found (or we are ignoring an existing plan) for .META.,,1.1028785192 so generated a random one; hri=.META.,,1.1028785192, src=, dest=asf001.sp2.ygridcore.net,60884,1365336878389; 1 (online=1, available=1) available servers, forceNewPlan=false 2013-04-07 12:14:42,825 INFO [MASTER_META_SERVER_OPERATIONS-asf001.sp2.ygridcore.net,33796,1365336878036-0] master.RegionStates(264): Region {NAME => '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192,} transitioned from {.META.,,1.1028785192 state=OFFLINE, ts=1365336881807, server=null} to {.META.,,1.1028785192 state=OFFLINE, ts=1365336882825, server=null} 2013-04-07 12:14:42,825 DEBUG [MASTER_META_SERVER_OPERATIONS-asf001.sp2.ygridcore.net,33796,1365336878036-0] zookeeper.ZKAssign(208): master:33796-0x13de46b5f8b0000 Creating (or updating) unassigned node for 1028785192 with OFFLINE state 2013-04-07 12:14:42,847 DEBUG [RegionServer:0;asf001.sp2.ygridcore.net,60884,1365336878389-EventThread] zookeeper.ZKUtil(424): regionserver:60884-0x13de46b5f8b0001 /hbase/meta-region-server does not exist. Watcher is set. 2013-04-07 12:14:42,847 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZKUtil(424): master:33796-0x13de46b5f8b0000 /hbase/meta-region-server does not exist. Watcher is set. 2013-04-07 12:14:42,850 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(274): master:33796-0x13de46b5f8b0000 Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/region-in-transition 2013-04-07 12:14:42,851 INFO [MASTER_META_SERVER_OPERATIONS-asf001.sp2.ygridcore.net,33796,1365336878036-0] master.AssignmentManager(1780): Assigning region .META.,,1.1028785192 to asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:14:42,852 INFO [MASTER_META_SERVER_OPERATIONS-asf001.sp2.ygridcore.net,33796,1365336878036-0] master.RegionStates(264): Region {NAME => '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192,} transitioned from {.META.,,1.1028785192 state=OFFLINE, ts=1365336882825, server=null} to {.META.,,1.1028785192 state=PENDING_OPEN, ts=1365336882852, server=asf001.sp2.ygridcore.net,60884,1365336878389} 2013-04-07 12:14:42,852 DEBUG [MASTER_META_SERVER_OPERATIONS-asf001.sp2.ygridcore.net,33796,1365336878036-0] master.ServerManager(713): New connection to asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:14:42,854 INFO [PRI IPC Server handler 0 on 60884] regionserver.HRegionServer(3346): Received request to open region: .META.,,1.1028785192 on asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:14:42,856 DEBUG [RS_OPEN_META-asf001.sp2.ygridcore.net,60884,1365336878389-0] zookeeper.ZKAssign(721): regionserver:60884-0x13de46b5f8b0001 Attempting to transition node 1028785192/.META. from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING 2013-04-07 12:14:42,856 INFO [MASTER_META_SERVER_OPERATIONS-asf001.sp2.ygridcore.net,33796,1365336878036-0] handler.ServerShutdownHandler(112): Splitting logs for asf001.sp2.ygridcore.net,35684,1365336878445 2013-04-07 12:14:42,858 DEBUG [RS_OPEN_META-asf001.sp2.ygridcore.net,60884,1365336878389-0] zookeeper.ZKUtil(1680): regionserver:60884-0x13de46b5f8b0001 Retrieved 65 byte(s) of data from znode /hbase/region-in-transition/1028785192; data=region=.META.,,1, state=M_ZK_REGION_OFFLINE, servername=asf001.sp2.ygridcore.net,60884,1365336878389, createTime=1365336882845, payload.length=0 2013-04-07 12:14:42,859 INFO [MASTER_META_SERVER_OPERATIONS-asf001.sp2.ygridcore.net,33796,1365336878036-0] master.SplitLogManager(959): dead splitlog workers [asf001.sp2.ygridcore.net,35684,1365336878445] 2013-04-07 12:14:42,862 DEBUG [MASTER_META_SERVER_OPERATIONS-asf001.sp2.ygridcore.net,33796,1365336878036-0] master.SplitLogManager(268): Scheduling batch of logs to split 2013-04-07 12:14:42,862 INFO [MASTER_META_SERVER_OPERATIONS-asf001.sp2.ygridcore.net,33796,1365336878036-0] master.SplitLogManager(270): started splitting logs in [hdfs://localhost:60405/user/jenkins/hbase/.logs/asf001.sp2.ygridcore.net,35684,1365336878445-splitting] 2013-04-07 12:14:42,862 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(274): master:33796-0x13de46b5f8b0000 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/region-in-transition/1028785192 2013-04-07 12:14:42,862 DEBUG [RS_OPEN_META-asf001.sp2.ygridcore.net,60884,1365336878389-0] zookeeper.ZKAssign(784): regionserver:60884-0x13de46b5f8b0001 Successfully transitioned node 1028785192 from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING 2013-04-07 12:14:42,862 DEBUG [RS_OPEN_META-asf001.sp2.ygridcore.net,60884,1365336878389-0] regionserver.HRegionServer(1319): logdir=hdfs://localhost:60405/user/jenkins/hbase/.logs/asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:14:42,863 INFO [RS_OPEN_META-asf001.sp2.ygridcore.net,60884,1365336878389-0] wal.FSHLog(354): HLog configuration: blocksize=64 MB, rollsize=60.8 MB, enabled=true, optionallogflushinternal=1000ms 2013-04-07 12:14:42,865 DEBUG [RegionServer:0;asf001.sp2.ygridcore.net,60884,1365336878389-EventThread] zookeeper.ZooKeeperWatcher(274): regionserver:60884-0x13de46b5f8b0001 Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/splitlog 2013-04-07 12:14:42,865 DEBUG [RegionServer:0;asf001.sp2.ygridcore.net,60884,1365336878389-EventThread] regionserver.SplitLogWorker(506): tasks arrived or departed 2013-04-07 12:14:42,865 DEBUG [pool-1-thread-1-EventThread] master.SplitLogManager(462): put up splitlog task at znode /hbase/splitlog/.logs%2Fasf001.sp2.ygridcore.net%2C35684%2C1365336878445-splitting%2Fasf001.sp2.ygridcore.net%252C35684%252C1365336878445.1365336879032 2013-04-07 12:14:42,867 DEBUG [hbase-am-zkevent-worker-pool-2-thread-6] zookeeper.ZKUtil(1680): master:33796-0x13de46b5f8b0000 Retrieved 63 byte(s) of data from znode /hbase/region-in-transition/1028785192 and set watcher; region=.META.,,1, state=RS_ZK_REGION_OPENING, servername=asf001.sp2.ygridcore.net,60884,1365336878389, createTime=1365336882858, payload.length=0 2013-04-07 12:14:42,868 DEBUG [hbase-am-zkevent-worker-pool-2-thread-6] master.AssignmentManager(740): Handling transition=RS_ZK_REGION_OPENING, server=asf001.sp2.ygridcore.net,60884,1365336878389, region=1028785192/.META., current state from region state map ={.META.,,1.1028785192 state=PENDING_OPEN, ts=1365336882852, server=asf001.sp2.ygridcore.net,60884,1365336878389} 2013-04-07 12:14:42,868 INFO [hbase-am-zkevent-worker-pool-2-thread-6] master.RegionStates(264): Region {NAME => '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192,} transitioned from {.META.,,1.1028785192 state=PENDING_OPEN, ts=1365336882852, server=asf001.sp2.ygridcore.net,60884,1365336878389} to {.META.,,1.1028785192 state=OPENING, ts=1365336882868, server=asf001.sp2.ygridcore.net,60884,1365336878389} 2013-04-07 12:14:42,868 DEBUG [pool-1-thread-1-EventThread] master.SplitLogManager(504): task not yet acquired /hbase/splitlog/.logs%2Fasf001.sp2.ygridcore.net%2C35684%2C1365336878445-splitting%2Fasf001.sp2.ygridcore.net%252C35684%252C1365336878445.1365336879032 ver = 0 2013-04-07 12:14:42,869 DEBUG [RS_OPEN_META-asf001.sp2.ygridcore.net,60884,1365336878389-0] wal.SequenceFileLogWriter(189): using new createWriter -- HADOOP-6840 2013-04-07 12:14:42,869 DEBUG [RS_OPEN_META-asf001.sp2.ygridcore.net,60884,1365336878389-0] wal.SequenceFileLogWriter(193): Path=hdfs://localhost:60405/user/jenkins/hbase/.logs/asf001.sp2.ygridcore.net,60884,1365336878389/asf001.sp2.ygridcore.net%2C60884%2C1365336878389.1365336882865.meta, compression=false 2013-04-07 12:14:42,869 INFO [RS_OPEN_META-asf001.sp2.ygridcore.net,60884,1365336878389-0] wal.FSHLog(552): Rolled log; new path=/user/jenkins/hbase/.logs/asf001.sp2.ygridcore.net,60884,1365336878389/asf001.sp2.ygridcore.net%2C60884%2C1365336878389.1365336882865.meta 2013-04-07 12:14:42,870 INFO [RS_OPEN_META-asf001.sp2.ygridcore.net,60884,1365336878389-0] wal.FSHLog(448): Using getNumCurrentReplicas--HDFS-826 2013-04-07 12:14:42,870 DEBUG [SplitLogWorker-asf001.sp2.ygridcore.net,60884,1365336878389] zookeeper.ZKUtil(1680): regionserver:60884-0x13de46b5f8b0001 Retrieved 45 byte(s) of data from znode /hbase/splitlog/.logs%2Fasf001.sp2.ygridcore.net%2C35684%2C1365336878445-splitting%2Fasf001.sp2.ygridcore.net%252C35684%252C1365336878445.1365336879032; data=PBUF\x08\x00\x12%\x0A\x18asf0... 2013-04-07 12:14:42,870 INFO [RS_OPEN_META-asf001.sp2.ygridcore.net,60884,1365336878389-0] regionserver.HRegion(4023): HRegion.openHRegion Region name ==.META.,,1.1028785192 2013-04-07 12:14:42,870 DEBUG [RS_OPEN_META-asf001.sp2.ygridcore.net,60884,1365336878389-0] regionserver.HRegion(4025): Opening region: {NAME => '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192,} 2013-04-07 12:14:42,871 DEBUG [RS_OPEN_META-asf001.sp2.ygridcore.net,60884,1365336878389-0] coprocessor.CoprocessorHost(182): Loading coprocessor class org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint with path null and priority 536870911 2013-04-07 12:14:42,871 DEBUG [RS_OPEN_META-asf001.sp2.ygridcore.net,60884,1365336878389-0] regionserver.HRegion(5025): Registered coprocessor service: region=.META.,,1 service=MultiRowMutationService 2013-04-07 12:14:42,872 INFO [RS_OPEN_META-asf001.sp2.ygridcore.net,60884,1365336878389-0] regionserver.RegionCoprocessorHost(193): Load coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint from HTD of .META. successfully. 2013-04-07 12:14:42,873 DEBUG [RS_OPEN_META-asf001.sp2.ygridcore.net,60884,1365336878389-0] regionserver.MetricsRegionSourceImpl(54): Creating new MetricsRegionSourceImpl for table .META. 1028785192 2013-04-07 12:14:42,873 DEBUG [RS_OPEN_META-asf001.sp2.ygridcore.net,60884,1365336878389-0] regionserver.HRegion(496): Instantiated .META.,,1.1028785192 2013-04-07 12:14:42,879 INFO [StoreOpenerThread-.META.,,1.1028785192-1] compactions.CompactionConfiguration(85): Compaction configuration size [134217728, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; delete expired; major period 86400000, major jitter 0.200000 2013-04-07 12:14:42,884 INFO [RS_OPEN_META-asf001.sp2.ygridcore.net,60884,1365336878389-0] regionserver.HRegion(586): Onlined .META.,,1.1028785192; next sequenceid=1 2013-04-07 12:14:42,885 DEBUG [RS_OPEN_META-asf001.sp2.ygridcore.net,60884,1365336878389-0] zookeeper.ZKAssign(721): regionserver:60884-0x13de46b5f8b0001 Attempting to transition node 1028785192/.META. from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENING 2013-04-07 12:14:43,000 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(274): master:33796-0x13de46b5f8b0000 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/splitlog/.logs%2Fasf001.sp2.ygridcore.net%2C35684%2C1365336878445-splitting%2Fasf001.sp2.ygridcore.net%252C35684%252C1365336878445.1365336879032 2013-04-07 12:14:43,000 INFO [SplitLogWorker-asf001.sp2.ygridcore.net,60884,1365336878389] regionserver.SplitLogWorker(267): worker asf001.sp2.ygridcore.net,60884,1365336878389 acquired task /hbase/splitlog/.logs%2Fasf001.sp2.ygridcore.net%2C35684%2C1365336878445-splitting%2Fasf001.sp2.ygridcore.net%252C35684%252C1365336878445.1365336879032 2013-04-07 12:14:43,002 DEBUG [RS_OPEN_META-asf001.sp2.ygridcore.net,60884,1365336878389-0] zookeeper.ZKUtil(1680): regionserver:60884-0x13de46b5f8b0001 Retrieved 63 byte(s) of data from znode /hbase/region-in-transition/1028785192; data=region=.META.,,1, state=RS_ZK_REGION_OPENING, servername=asf001.sp2.ygridcore.net,60884,1365336878389, createTime=1365336882858, payload.length=0 2013-04-07 12:14:43,002 INFO [SplitLogWorker-asf001.sp2.ygridcore.net,60884,1365336878389] wal.HLogSplitter(432): Splitting hlog: hdfs://localhost:60405/user/jenkins/hbase/.logs/asf001.sp2.ygridcore.net,35684,1365336878445-splitting/asf001.sp2.ygridcore.net%2C35684%2C1365336878445.1365336879032, length=0 2013-04-07 12:14:43,003 INFO [pool-1-thread-1-EventThread] master.SplitLogManager(577): task /hbase/splitlog/.logs%2Fasf001.sp2.ygridcore.net%2C35684%2C1365336878445-splitting%2Fasf001.sp2.ygridcore.net%252C35684%252C1365336878445.1365336879032 acquired by asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:14:43,003 WARN [SplitLogWorker-asf001.sp2.ygridcore.net,60884,1365336878389] wal.HLogSplitter(739): File hdfs://localhost:60405/user/jenkins/hbase/.logs/asf001.sp2.ygridcore.net,35684,1365336878445-splitting/asf001.sp2.ygridcore.net%2C35684%2C1365336878445.1365336879032 might be still open, length is 0 2013-04-07 12:14:43,003 INFO [SplitLogWorker-asf001.sp2.ygridcore.net,60884,1365336878389] util.FSHDFSUtils(56): Recovering file hdfs://localhost:60405/user/jenkins/hbase/.logs/asf001.sp2.ygridcore.net,35684,1365336878445-splitting/asf001.sp2.ygridcore.net%2C35684%2C1365336878445.1365336879032 2013-04-07 12:14:43,004 WARN [IPC Server handler 1 on 60405] namenode.FSNamesystem(2321): BLOCK* internalReleaseLease: No blocks found, lease removed for /user/jenkins/hbase/.logs/asf001.sp2.ygridcore.net,35684,1365336878445-splitting/asf001.sp2.ygridcore.net%2C35684%2C1365336878445.1365336879032 2013-04-07 12:14:43,004 INFO [SplitLogWorker-asf001.sp2.ygridcore.net,60884,1365336878389] util.FSHDFSUtils(72): Attempt 1 to recoverLease on file hdfs://localhost:60405/user/jenkins/hbase/.logs/asf001.sp2.ygridcore.net,35684,1365336878445-splitting/asf001.sp2.ygridcore.net%2C35684%2C1365336878445.1365336879032 returned false, trying for 1ms 2013-04-07 12:14:43,005 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(274): master:33796-0x13de46b5f8b0000 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/region-in-transition/1028785192 2013-04-07 12:14:43,006 DEBUG [RS_OPEN_META-asf001.sp2.ygridcore.net,60884,1365336878389-0] zookeeper.ZKAssign(784): regionserver:60884-0x13de46b5f8b0001 Successfully transitioned node 1028785192 from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENING 2013-04-07 12:14:43,006 INFO [PostOpenDeployTasks:1028785192] regionserver.HRegionServer(1563): Post open deploy tasks for region=.META.,,1.1028785192 2013-04-07 12:14:43,006 INFO [PostOpenDeployTasks:1028785192] zookeeper.MetaRegionTracker(122): Setting META region location in ZooKeeper as asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:14:43,007 DEBUG [hbase-am-zkevent-worker-pool-2-thread-7] zookeeper.ZKUtil(1680): master:33796-0x13de46b5f8b0000 Retrieved 63 byte(s) of data from znode /hbase/region-in-transition/1028785192 and set watcher; region=.META.,,1, state=RS_ZK_REGION_OPENING, servername=asf001.sp2.ygridcore.net,60884,1365336878389, createTime=1365336883003, payload.length=0 2013-04-07 12:14:43,007 DEBUG [hbase-am-zkevent-worker-pool-2-thread-7] master.AssignmentManager(740): Handling transition=RS_ZK_REGION_OPENING, server=asf001.sp2.ygridcore.net,60884,1365336878389, region=1028785192/.META., current state from region state map ={.META.,,1.1028785192 state=OPENING, ts=1365336882868, server=asf001.sp2.ygridcore.net,60884,1365336878389} 2013-04-07 12:14:43,007 INFO [hbase-am-zkevent-worker-pool-2-thread-7] master.RegionStates(264): Region {NAME => '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192,} transitioned from {.META.,,1.1028785192 state=OPENING, ts=1365336882868, server=asf001.sp2.ygridcore.net,60884,1365336878389} to {.META.,,1.1028785192 state=OPENING, ts=1365336883007, server=asf001.sp2.ygridcore.net,60884,1365336878389} 2013-04-07 12:14:43,010 DEBUG [RegionServer:0;asf001.sp2.ygridcore.net,60884,1365336878389-EventThread] zookeeper.ZooKeeperWatcher(274): regionserver:60884-0x13de46b5f8b0001 Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/meta-region-server 2013-04-07 12:14:43,010 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(274): master:33796-0x13de46b5f8b0000 Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/meta-region-server 2013-04-07 12:14:43,011 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZKUtil(1680): master:33796-0x13de46b5f8b0000 Retrieved 43 byte(s) of data from znode /hbase/meta-region-server and set watcher; asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:14:43,012 DEBUG [RegionServer:0;asf001.sp2.ygridcore.net,60884,1365336878389-EventThread] zookeeper.ZKUtil(1680): regionserver:60884-0x13de46b5f8b0001 Retrieved 43 byte(s) of data from znode /hbase/meta-region-server and set watcher; asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:14:43,012 INFO [PostOpenDeployTasks:1028785192] regionserver.HRegionServer(1584): Done with post open deploy task for region=.META.,,1.1028785192 2013-04-07 12:14:43,012 DEBUG [RS_OPEN_META-asf001.sp2.ygridcore.net,60884,1365336878389-0] zookeeper.ZKAssign(721): regionserver:60884-0x13de46b5f8b0001 Attempting to transition node 1028785192/.META. from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENED 2013-04-07 12:14:43,014 DEBUG [RS_OPEN_META-asf001.sp2.ygridcore.net,60884,1365336878389-0] zookeeper.ZKUtil(1680): regionserver:60884-0x13de46b5f8b0001 Retrieved 63 byte(s) of data from znode /hbase/region-in-transition/1028785192; data=region=.META.,,1, state=RS_ZK_REGION_OPENING, servername=asf001.sp2.ygridcore.net,60884,1365336878389, createTime=1365336883003, payload.length=0 2013-04-07 12:14:43,016 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(274): master:33796-0x13de46b5f8b0000 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/region-in-transition/1028785192 2013-04-07 12:14:43,016 DEBUG [RS_OPEN_META-asf001.sp2.ygridcore.net,60884,1365336878389-0] zookeeper.ZKAssign(784): regionserver:60884-0x13de46b5f8b0001 Successfully transitioned node 1028785192 from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENED 2013-04-07 12:14:43,016 DEBUG [RS_OPEN_META-asf001.sp2.ygridcore.net,60884,1365336878389-0] handler.OpenRegionHandler(361): region transitioned to opened in zookeeper: {NAME => '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192,}, server: asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:14:43,016 DEBUG [RS_OPEN_META-asf001.sp2.ygridcore.net,60884,1365336878389-0] handler.OpenRegionHandler(168): Opened .META.,,1.1028785192 on server:asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:14:43,017 DEBUG [hbase-am-zkevent-worker-pool-2-thread-8] zookeeper.ZKUtil(1680): master:33796-0x13de46b5f8b0000 Retrieved 63 byte(s) of data from znode /hbase/region-in-transition/1028785192 and set watcher; region=.META.,,1, state=RS_ZK_REGION_OPENED, servername=asf001.sp2.ygridcore.net,60884,1365336878389, createTime=1365336883014, payload.length=0 2013-04-07 12:14:43,017 DEBUG [hbase-am-zkevent-worker-pool-2-thread-8] master.AssignmentManager(740): Handling transition=RS_ZK_REGION_OPENED, server=asf001.sp2.ygridcore.net,60884,1365336878389, region=1028785192/.META., current state from region state map ={.META.,,1.1028785192 state=OPENING, ts=1365336883007, server=asf001.sp2.ygridcore.net,60884,1365336878389} 2013-04-07 12:14:43,017 INFO [hbase-am-zkevent-worker-pool-2-thread-8] master.RegionStates(264): Region {NAME => '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192,} transitioned from {.META.,,1.1028785192 state=OPENING, ts=1365336883007, server=asf001.sp2.ygridcore.net,60884,1365336878389} to {.META.,,1.1028785192 state=OPEN, ts=1365336883017, server=asf001.sp2.ygridcore.net,60884,1365336878389} 2013-04-07 12:14:43,018 INFO [MASTER_OPEN_REGION-asf001.sp2.ygridcore.net,33796,1365336878036-1] handler.OpenedRegionHandler(144): Handling OPENED event for .META.,,1.1028785192 from asf001.sp2.ygridcore.net,60884,1365336878389; deleting unassigned node 2013-04-07 12:14:43,018 DEBUG [MASTER_OPEN_REGION-asf001.sp2.ygridcore.net,33796,1365336878036-1] zookeeper.ZKAssign(405): master:33796-0x13de46b5f8b0000 Deleting existing unassigned node for 1028785192 that is in expected state RS_ZK_REGION_OPENED 2013-04-07 12:14:43,020 DEBUG [MASTER_OPEN_REGION-asf001.sp2.ygridcore.net,33796,1365336878036-1] zookeeper.ZKUtil(1680): master:33796-0x13de46b5f8b0000 Retrieved 63 byte(s) of data from znode /hbase/region-in-transition/1028785192; data=region=.META.,,1, state=RS_ZK_REGION_OPENED, servername=asf001.sp2.ygridcore.net,60884,1365336878389, createTime=1365336883014, payload.length=0 2013-04-07 12:14:43,023 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(274): master:33796-0x13de46b5f8b0000 Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/region-in-transition/1028785192 2013-04-07 12:14:43,023 DEBUG [hbase-am-zkevent-worker-pool-2-thread-9] master.AssignmentManager$4(1159): The znode of region .META.,,1.1028785192 has been deleted. 2013-04-07 12:14:43,023 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(274): master:33796-0x13de46b5f8b0000 Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/region-in-transition 2013-04-07 12:14:43,024 INFO [hbase-am-zkevent-worker-pool-2-thread-9] master.RegionStates(264): Region {NAME => '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192,} transitioned from {.META.,,1.1028785192 state=OPEN, ts=1365336883017, server=asf001.sp2.ygridcore.net,60884,1365336878389} to {.META.,,1.1028785192 state=OPEN, ts=1365336883024, server=asf001.sp2.ygridcore.net,60884,1365336878389} 2013-04-07 12:14:43,024 INFO [hbase-am-zkevent-worker-pool-2-thread-9] master.RegionStates(299): Onlined region {NAME => '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192,} on asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:14:43,024 DEBUG [MASTER_OPEN_REGION-asf001.sp2.ygridcore.net,33796,1365336878036-1] zookeeper.ZKAssign(434): master:33796-0x13de46b5f8b0000 Successfully deleted unassigned node for region 1028785192 in expected state RS_ZK_REGION_OPENED 2013-04-07 12:14:43,024 INFO [hbase-am-zkevent-worker-pool-2-thread-9] master.AssignmentManager$4(1164): The master has opened the region .META.,,1.1028785192 that was online on asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:14:43,056 INFO [RegionServer:1;asf001.sp2.ygridcore.net,35684,1365336878445.leaseChecker] regionserver.Leases(124): RegionServer:1;asf001.sp2.ygridcore.net,35684,1365336878445.leaseChecker closing leases 2013-04-07 12:14:43,056 INFO [RegionServer:1;asf001.sp2.ygridcore.net,35684,1365336878445.leaseChecker] regionserver.Leases(131): RegionServer:1;asf001.sp2.ygridcore.net,35684,1365336878445.leaseChecker closed leases 2013-04-07 12:14:43,390 DEBUG [MASTER_TABLE_OPERATIONS-asf001.sp2.ygridcore.net,33796,1365336878036-0] zookeeper.ZKUtil(1680): hconnection-0x1f327e-0x13de46b5f8b0003 Retrieved 43 byte(s) of data from znode /hbase/meta-region-server; data=asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:14:43,412 INFO [MASTER_TABLE_OPERATIONS-asf001.sp2.ygridcore.net,33796,1365336878036-0] catalog.MetaEditor(254): Added 1 regions in META 2013-04-07 12:14:43,413 INFO [MASTER_TABLE_OPERATIONS-asf001.sp2.ygridcore.net,33796,1365336878036-0] master.AssignmentManager(2363): Not use bulk assigning since we are assigning only 1 region(s) to 1 server(s) 2013-04-07 12:14:43,413 DEBUG [MASTER_TABLE_OPERATIONS-asf001.sp2.ygridcore.net,33796,1365336878036-0] master.AssignmentManager(1444): Bulk assigning 1 region(s) to asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:14:43,415 INFO [MASTER_TABLE_OPERATIONS-asf001.sp2.ygridcore.net,33796,1365336878036-0] master.RegionStates(264): Region {NAME => 'test1365336881340,,1365336881424.c9e9e308a02801bb891879b6cf471381.', STARTKEY => '', ENDKEY => '', ENCODED => c9e9e308a02801bb891879b6cf471381,} transitioned from {test1365336881340,,1365336881424.c9e9e308a02801bb891879b6cf471381. state=OFFLINE, ts=1365336883413, server=null} to {test1365336881340,,1365336881424.c9e9e308a02801bb891879b6cf471381. state=OFFLINE, ts=1365336883415, server=null} 2013-04-07 12:14:43,415 DEBUG [MASTER_TABLE_OPERATIONS-asf001.sp2.ygridcore.net,33796,1365336878036-0] zookeeper.ZKAssign(177): master:33796-0x13de46b5f8b0000 Async create of unassigned node for c9e9e308a02801bb891879b6cf471381 with OFFLINE state 2013-04-07 12:14:43,417 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(274): master:33796-0x13de46b5f8b0000 Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/region-in-transition 2013-04-07 12:14:43,417 DEBUG [pool-1-thread-1-EventThread] master.OfflineCallback(69): rs={test1365336881340,,1365336881424.c9e9e308a02801bb891879b6cf471381. state=OFFLINE, ts=1365336883413, server=null}, server=asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:14:43,419 DEBUG [pool-1-thread-1-EventThread] master.OfflineCallback$ExistCallback(106): rs={test1365336881340,,1365336881424.c9e9e308a02801bb891879b6cf471381. state=OFFLINE, ts=1365336883413, server=null}, server=asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:14:43,420 INFO [MASTER_TABLE_OPERATIONS-asf001.sp2.ygridcore.net,33796,1365336878036-0] master.AssignmentManager(1482): asf001.sp2.ygridcore.net,60884,1365336878389 unassigned znodes=1 of total=1 2013-04-07 12:14:43,420 INFO [MASTER_TABLE_OPERATIONS-asf001.sp2.ygridcore.net,33796,1365336878036-0] master.RegionStates(264): Region {NAME => 'test1365336881340,,1365336881424.c9e9e308a02801bb891879b6cf471381.', STARTKEY => '', ENDKEY => '', ENCODED => c9e9e308a02801bb891879b6cf471381,} transitioned from {test1365336881340,,1365336881424.c9e9e308a02801bb891879b6cf471381. state=OFFLINE, ts=1365336883415, server=null} to {test1365336881340,,1365336881424.c9e9e308a02801bb891879b6cf471381. state=PENDING_OPEN, ts=1365336883420, server=asf001.sp2.ygridcore.net,60884,1365336878389} 2013-04-07 12:14:43,422 INFO [PRI IPC Server handler 2 on 60884] regionserver.HRegionServer(3346): Received request to open region: test1365336881340,,1365336881424.c9e9e308a02801bb891879b6cf471381. on asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:14:43,428 DEBUG [RS_OPEN_REGION-asf001.sp2.ygridcore.net,60884,1365336878389-0] zookeeper.ZKAssign(721): regionserver:60884-0x13de46b5f8b0001 Attempting to transition node c9e9e308a02801bb891879b6cf471381 from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING 2013-04-07 12:14:43,428 DEBUG [MASTER_TABLE_OPERATIONS-asf001.sp2.ygridcore.net,33796,1365336878036-0] master.AssignmentManager(1600): Bulk assigning done for asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:14:43,430 DEBUG [RS_OPEN_REGION-asf001.sp2.ygridcore.net,60884,1365336878389-0] zookeeper.ZKUtil(1680): regionserver:60884-0x13de46b5f8b0001 Retrieved 120 byte(s) of data from znode /hbase/region-in-transition/c9e9e308a02801bb891879b6cf471381; data=region=test1365336881340,,1365336881424.c9e9e308a02801bb891879b6cf471381., state=M_ZK_REGION_OFFLINE, servername=asf001.sp2.ygridcore.net,60884,1365336878389, createTime=1365336883415, payload.length=0 2013-04-07 12:14:43,431 DEBUG [MASTER_TABLE_OPERATIONS-asf001.sp2.ygridcore.net,33796,1365336878036-0] master.TableLockManager$ZKTableLockManager$TableLockImpl(253): Attempt to release table write lock on :test1365336881340 2013-04-07 12:14:43,433 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(274): master:33796-0x13de46b5f8b0000 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/region-in-transition/c9e9e308a02801bb891879b6cf471381 2013-04-07 12:14:43,433 DEBUG [RS_OPEN_REGION-asf001.sp2.ygridcore.net,60884,1365336878389-0] zookeeper.ZKAssign(784): regionserver:60884-0x13de46b5f8b0001 Successfully transitioned node c9e9e308a02801bb891879b6cf471381 from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING 2013-04-07 12:14:43,433 INFO [RS_OPEN_REGION-asf001.sp2.ygridcore.net,60884,1365336878389-0] regionserver.HRegion(4023): HRegion.openHRegion Region name ==test1365336881340,,1365336881424.c9e9e308a02801bb891879b6cf471381. 2013-04-07 12:14:43,433 DEBUG [RS_OPEN_REGION-asf001.sp2.ygridcore.net,60884,1365336878389-0] regionserver.HRegion(4025): Opening region: {NAME => 'test1365336881340,,1365336881424.c9e9e308a02801bb891879b6cf471381.', STARTKEY => '', ENDKEY => '', ENCODED => c9e9e308a02801bb891879b6cf471381,} 2013-04-07 12:14:43,436 DEBUG [hbase-am-zkevent-worker-pool-2-thread-12] zookeeper.ZKUtil(1680): master:33796-0x13de46b5f8b0000 Retrieved 120 byte(s) of data from znode /hbase/region-in-transition/c9e9e308a02801bb891879b6cf471381 and set watcher; region=test1365336881340,,1365336881424.c9e9e308a02801bb891879b6cf471381., state=RS_ZK_REGION_OPENING, servername=asf001.sp2.ygridcore.net,60884,1365336878389, createTime=1365336883431, payload.length=0 2013-04-07 12:14:43,436 DEBUG [hbase-am-zkevent-worker-pool-2-thread-12] master.AssignmentManager(740): Handling transition=RS_ZK_REGION_OPENING, server=asf001.sp2.ygridcore.net,60884,1365336878389, region=c9e9e308a02801bb891879b6cf471381, current state from region state map ={test1365336881340,,1365336881424.c9e9e308a02801bb891879b6cf471381. state=PENDING_OPEN, ts=1365336883420, server=asf001.sp2.ygridcore.net,60884,1365336878389} 2013-04-07 12:14:43,436 INFO [hbase-am-zkevent-worker-pool-2-thread-12] master.RegionStates(264): Region {NAME => 'test1365336881340,,1365336881424.c9e9e308a02801bb891879b6cf471381.', STARTKEY => '', ENDKEY => '', ENCODED => c9e9e308a02801bb891879b6cf471381,} transitioned from {test1365336881340,,1365336881424.c9e9e308a02801bb891879b6cf471381. state=PENDING_OPEN, ts=1365336883420, server=asf001.sp2.ygridcore.net,60884,1365336878389} to {test1365336881340,,1365336881424.c9e9e308a02801bb891879b6cf471381. state=OPENING, ts=1365336883436, server=asf001.sp2.ygridcore.net,60884,1365336878389} 2013-04-07 12:14:43,436 DEBUG [RS_OPEN_REGION-asf001.sp2.ygridcore.net,60884,1365336878389-0] regionserver.MetricsRegionSourceImpl(54): Creating new MetricsRegionSourceImpl for table test1365336881340 c9e9e308a02801bb891879b6cf471381 2013-04-07 12:14:43,437 DEBUG [RS_OPEN_REGION-asf001.sp2.ygridcore.net,60884,1365336878389-0] regionserver.HRegion(496): Instantiated test1365336881340,,1365336881424.c9e9e308a02801bb891879b6cf471381. 2013-04-07 12:14:43,437 DEBUG [MASTER_TABLE_OPERATIONS-asf001.sp2.ygridcore.net,33796,1365336878036-0] lock.ZKInterProcessLockBase(297): Successfully released /hbase/table-lock/test1365336881340/write-master:337960000000000 2013-04-07 12:14:43,438 DEBUG [MASTER_TABLE_OPERATIONS-asf001.sp2.ygridcore.net,33796,1365336878036-0] master.TableLockManager$ZKTableLockManager$TableLockImpl(269): Released table lock on :test1365336881340 2013-04-07 12:14:43,445 INFO [StoreOpenerThread-test1365336881340,,1365336881424.c9e9e308a02801bb891879b6cf471381.-1] compactions.CompactionConfiguration(85): Compaction configuration size [134217728, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; delete expired; major period 86400000, major jitter 0.200000 2013-04-07 12:14:43,449 INFO [RS_OPEN_REGION-asf001.sp2.ygridcore.net,60884,1365336878389-0] regionserver.HRegion(586): Onlined test1365336881340,,1365336881424.c9e9e308a02801bb891879b6cf471381.; next sequenceid=1 2013-04-07 12:14:43,449 DEBUG [RS_OPEN_REGION-asf001.sp2.ygridcore.net,60884,1365336878389-0] zookeeper.ZKAssign(721): regionserver:60884-0x13de46b5f8b0001 Attempting to transition node c9e9e308a02801bb891879b6cf471381 from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENING 2013-04-07 12:14:43,451 DEBUG [RS_OPEN_REGION-asf001.sp2.ygridcore.net,60884,1365336878389-0] zookeeper.ZKUtil(1680): regionserver:60884-0x13de46b5f8b0001 Retrieved 120 byte(s) of data from znode /hbase/region-in-transition/c9e9e308a02801bb891879b6cf471381; data=region=test1365336881340,,1365336881424.c9e9e308a02801bb891879b6cf471381., state=RS_ZK_REGION_OPENING, servername=asf001.sp2.ygridcore.net,60884,1365336878389, createTime=1365336883431, payload.length=0 2013-04-07 12:14:43,452 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(274): master:33796-0x13de46b5f8b0000 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/region-in-transition/c9e9e308a02801bb891879b6cf471381 2013-04-07 12:14:43,453 DEBUG [RS_OPEN_REGION-asf001.sp2.ygridcore.net,60884,1365336878389-0] zookeeper.ZKAssign(784): regionserver:60884-0x13de46b5f8b0001 Successfully transitioned node c9e9e308a02801bb891879b6cf471381 from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENING 2013-04-07 12:14:43,453 INFO [PostOpenDeployTasks:c9e9e308a02801bb891879b6cf471381] regionserver.HRegionServer(1563): Post open deploy tasks for region=test1365336881340,,1365336881424.c9e9e308a02801bb891879b6cf471381. 2013-04-07 12:14:43,454 DEBUG [hbase-am-zkevent-worker-pool-2-thread-13] zookeeper.ZKUtil(1680): master:33796-0x13de46b5f8b0000 Retrieved 120 byte(s) of data from znode /hbase/region-in-transition/c9e9e308a02801bb891879b6cf471381 and set watcher; region=test1365336881340,,1365336881424.c9e9e308a02801bb891879b6cf471381., state=RS_ZK_REGION_OPENING, servername=asf001.sp2.ygridcore.net,60884,1365336878389, createTime=1365336883451, payload.length=0 2013-04-07 12:14:43,454 DEBUG [hbase-am-zkevent-worker-pool-2-thread-13] master.AssignmentManager(740): Handling transition=RS_ZK_REGION_OPENING, server=asf001.sp2.ygridcore.net,60884,1365336878389, region=c9e9e308a02801bb891879b6cf471381, current state from region state map ={test1365336881340,,1365336881424.c9e9e308a02801bb891879b6cf471381. state=OPENING, ts=1365336883436, server=asf001.sp2.ygridcore.net,60884,1365336878389} 2013-04-07 12:14:43,454 INFO [hbase-am-zkevent-worker-pool-2-thread-13] master.RegionStates(264): Region {NAME => 'test1365336881340,,1365336881424.c9e9e308a02801bb891879b6cf471381.', STARTKEY => '', ENDKEY => '', ENCODED => c9e9e308a02801bb891879b6cf471381,} transitioned from {test1365336881340,,1365336881424.c9e9e308a02801bb891879b6cf471381. state=OPENING, ts=1365336883436, server=asf001.sp2.ygridcore.net,60884,1365336878389} to {test1365336881340,,1365336881424.c9e9e308a02801bb891879b6cf471381. state=OPENING, ts=1365336883454, server=asf001.sp2.ygridcore.net,60884,1365336878389} 2013-04-07 12:14:43,454 DEBUG [PostOpenDeployTasks:c9e9e308a02801bb891879b6cf471381] zookeeper.ZKUtil(1680): hconnection-0x43ad4a-0x13de46b5f8b0005 Retrieved 43 byte(s) of data from znode /hbase/meta-region-server; data=asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:14:43,456 DEBUG [PostOpenDeployTasks:c9e9e308a02801bb891879b6cf471381] zookeeper.ZKUtil(1680): hconnection-0x43ad4a-0x13de46b5f8b0005 Retrieved 43 byte(s) of data from znode /hbase/meta-region-server; data=asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:14:43,462 INFO [PostOpenDeployTasks:c9e9e308a02801bb891879b6cf471381] catalog.MetaEditor(432): Updated row test1365336881340,,1365336881424.c9e9e308a02801bb891879b6cf471381. with server=asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:14:43,463 INFO [PostOpenDeployTasks:c9e9e308a02801bb891879b6cf471381] regionserver.HRegionServer(1584): Done with post open deploy task for region=test1365336881340,,1365336881424.c9e9e308a02801bb891879b6cf471381. 2013-04-07 12:14:43,463 DEBUG [RS_OPEN_REGION-asf001.sp2.ygridcore.net,60884,1365336878389-0] zookeeper.ZKAssign(721): regionserver:60884-0x13de46b5f8b0001 Attempting to transition node c9e9e308a02801bb891879b6cf471381 from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENED 2013-04-07 12:14:43,467 DEBUG [RS_OPEN_REGION-asf001.sp2.ygridcore.net,60884,1365336878389-0] zookeeper.ZKUtil(1680): regionserver:60884-0x13de46b5f8b0001 Retrieved 120 byte(s) of data from znode /hbase/region-in-transition/c9e9e308a02801bb891879b6cf471381; data=region=test1365336881340,,1365336881424.c9e9e308a02801bb891879b6cf471381., state=RS_ZK_REGION_OPENING, servername=asf001.sp2.ygridcore.net,60884,1365336878389, createTime=1365336883451, payload.length=0 2013-04-07 12:14:43,469 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(274): master:33796-0x13de46b5f8b0000 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/region-in-transition/c9e9e308a02801bb891879b6cf471381 2013-04-07 12:14:43,469 DEBUG [RS_OPEN_REGION-asf001.sp2.ygridcore.net,60884,1365336878389-0] zookeeper.ZKAssign(784): regionserver:60884-0x13de46b5f8b0001 Successfully transitioned node c9e9e308a02801bb891879b6cf471381 from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENED 2013-04-07 12:14:43,469 DEBUG [RS_OPEN_REGION-asf001.sp2.ygridcore.net,60884,1365336878389-0] handler.OpenRegionHandler(361): region transitioned to opened in zookeeper: {NAME => 'test1365336881340,,1365336881424.c9e9e308a02801bb891879b6cf471381.', STARTKEY => '', ENDKEY => '', ENCODED => c9e9e308a02801bb891879b6cf471381,}, server: asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:14:43,469 DEBUG [RS_OPEN_REGION-asf001.sp2.ygridcore.net,60884,1365336878389-0] handler.OpenRegionHandler(168): Opened test1365336881340,,1365336881424.c9e9e308a02801bb891879b6cf471381. on server:asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:14:43,470 DEBUG [hbase-am-zkevent-worker-pool-2-thread-14] zookeeper.ZKUtil(1680): master:33796-0x13de46b5f8b0000 Retrieved 120 byte(s) of data from znode /hbase/region-in-transition/c9e9e308a02801bb891879b6cf471381 and set watcher; region=test1365336881340,,1365336881424.c9e9e308a02801bb891879b6cf471381., state=RS_ZK_REGION_OPENED, servername=asf001.sp2.ygridcore.net,60884,1365336878389, createTime=1365336883467, payload.length=0 2013-04-07 12:14:43,470 DEBUG [hbase-am-zkevent-worker-pool-2-thread-14] master.AssignmentManager(740): Handling transition=RS_ZK_REGION_OPENED, server=asf001.sp2.ygridcore.net,60884,1365336878389, region=c9e9e308a02801bb891879b6cf471381, current state from region state map ={test1365336881340,,1365336881424.c9e9e308a02801bb891879b6cf471381. state=OPENING, ts=1365336883454, server=asf001.sp2.ygridcore.net,60884,1365336878389} 2013-04-07 12:14:43,471 INFO [hbase-am-zkevent-worker-pool-2-thread-14] master.RegionStates(264): Region {NAME => 'test1365336881340,,1365336881424.c9e9e308a02801bb891879b6cf471381.', STARTKEY => '', ENDKEY => '', ENCODED => c9e9e308a02801bb891879b6cf471381,} transitioned from {test1365336881340,,1365336881424.c9e9e308a02801bb891879b6cf471381. state=OPENING, ts=1365336883454, server=asf001.sp2.ygridcore.net,60884,1365336878389} to {test1365336881340,,1365336881424.c9e9e308a02801bb891879b6cf471381. state=OPEN, ts=1365336883471, server=asf001.sp2.ygridcore.net,60884,1365336878389} 2013-04-07 12:14:43,471 DEBUG [MASTER_OPEN_REGION-asf001.sp2.ygridcore.net,33796,1365336878036-2] handler.OpenedRegionHandler(146): Handling OPENED event for test1365336881340,,1365336881424.c9e9e308a02801bb891879b6cf471381. from asf001.sp2.ygridcore.net,60884,1365336878389; deleting unassigned node 2013-04-07 12:14:43,471 DEBUG [MASTER_OPEN_REGION-asf001.sp2.ygridcore.net,33796,1365336878036-2] zookeeper.ZKAssign(405): master:33796-0x13de46b5f8b0000 Deleting existing unassigned node for c9e9e308a02801bb891879b6cf471381 that is in expected state RS_ZK_REGION_OPENED 2013-04-07 12:14:43,473 DEBUG [MASTER_OPEN_REGION-asf001.sp2.ygridcore.net,33796,1365336878036-2] zookeeper.ZKUtil(1680): master:33796-0x13de46b5f8b0000 Retrieved 120 byte(s) of data from znode /hbase/region-in-transition/c9e9e308a02801bb891879b6cf471381; data=region=test1365336881340,,1365336881424.c9e9e308a02801bb891879b6cf471381., state=RS_ZK_REGION_OPENED, servername=asf001.sp2.ygridcore.net,60884,1365336878389, createTime=1365336883467, payload.length=0 2013-04-07 12:14:43,475 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(274): master:33796-0x13de46b5f8b0000 Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/region-in-transition/c9e9e308a02801bb891879b6cf471381 2013-04-07 12:14:43,475 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(274): master:33796-0x13de46b5f8b0000 Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/region-in-transition 2013-04-07 12:14:43,475 DEBUG [hbase-am-zkevent-worker-pool-2-thread-15] master.AssignmentManager$4(1159): The znode of region test1365336881340,,1365336881424.c9e9e308a02801bb891879b6cf471381. has been deleted. 2013-04-07 12:14:43,476 INFO [hbase-am-zkevent-worker-pool-2-thread-15] master.RegionStates(264): Region {NAME => 'test1365336881340,,1365336881424.c9e9e308a02801bb891879b6cf471381.', STARTKEY => '', ENDKEY => '', ENCODED => c9e9e308a02801bb891879b6cf471381,} transitioned from {test1365336881340,,1365336881424.c9e9e308a02801bb891879b6cf471381. state=OPEN, ts=1365336883471, server=asf001.sp2.ygridcore.net,60884,1365336878389} to {test1365336881340,,1365336881424.c9e9e308a02801bb891879b6cf471381. state=OPEN, ts=1365336883476, server=asf001.sp2.ygridcore.net,60884,1365336878389} 2013-04-07 12:14:43,476 INFO [hbase-am-zkevent-worker-pool-2-thread-15] master.RegionStates(299): Onlined region {NAME => 'test1365336881340,,1365336881424.c9e9e308a02801bb891879b6cf471381.', STARTKEY => '', ENDKEY => '', ENCODED => c9e9e308a02801bb891879b6cf471381,} on asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:14:43,476 DEBUG [MASTER_OPEN_REGION-asf001.sp2.ygridcore.net,33796,1365336878036-2] zookeeper.ZKAssign(434): master:33796-0x13de46b5f8b0000 Successfully deleted unassigned node for region c9e9e308a02801bb891879b6cf471381 in expected state RS_ZK_REGION_OPENED 2013-04-07 12:14:43,476 INFO [hbase-am-zkevent-worker-pool-2-thread-15] master.AssignmentManager$4(1164): The master has opened the region test1365336881340,,1365336881424.c9e9e308a02801bb891879b6cf471381. that was online on asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:14:43,477 DEBUG [asf001.sp2.ygridcore.net,33796,1365336878036.splitLogManagerTimeoutMonitor] master.SplitLogManager$TimeoutMonitor(1013): total tasks = 1 unassigned = 0 2013-04-07 12:14:43,504 DEBUG [pool-1-thread-1] zookeeper.ZKUtil(1680): hconnection-0xfe04c6-0x13de46b5f8b0007 Retrieved 6 byte(s) of data from znode /hbase/table/.META.; data=PBUF\x08\x00 2013-04-07 12:14:43,505 INFO [SplitLogWorker-asf001.sp2.ygridcore.net,60884,1365336878389] util.FSHDFSUtils(72): Attempt 2 to recoverLease on file hdfs://localhost:60405/user/jenkins/hbase/.logs/asf001.sp2.ygridcore.net,35684,1365336878445-splitting/asf001.sp2.ygridcore.net%2C35684%2C1365336878445.1365336879032 returned true, trying for 502ms 2013-04-07 12:14:43,505 DEBUG [pool-1-thread-1] zookeeper.ZKUtil(1680): hconnection-0xfe04c6-0x13de46b5f8b0007 Retrieved 43 byte(s) of data from znode /hbase/meta-region-server; data=asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:14:43,508 WARN [SplitLogWorker-asf001.sp2.ygridcore.net,60884,1365336878389] wal.HLogSplitter(753): Could not open hdfs://localhost:60405/user/jenkins/hbase/.logs/asf001.sp2.ygridcore.net,35684,1365336878445-splitting/asf001.sp2.ygridcore.net%2C35684%2C1365336878445.1365336879032 for reading. File is empty java.io.EOFException at java.io.DataInputStream.readFully(DataInputStream.java:180) at java.io.DataInputStream.readFully(DataInputStream.java:152) at org.apache.hadoop.io.SequenceFile$Reader.init(SequenceFile.java:1512) at org.apache.hadoop.io.SequenceFile$Reader.(SequenceFile.java:1490) at org.apache.hadoop.io.SequenceFile$Reader.(SequenceFile.java:1479) at org.apache.hadoop.io.SequenceFile$Reader.(SequenceFile.java:1474) at org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogReader$WALReader.(SequenceFileLogReader.java:56) at org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogReader.init(SequenceFileLogReader.java:178) at org.apache.hadoop.hbase.regionserver.wal.HLogFactory.createReader(HLogFactory.java:91) at org.apache.hadoop.hbase.regionserver.wal.HLogSplitter.getReader(HLogSplitter.java:832) at org.apache.hadoop.hbase.regionserver.wal.HLogSplitter.getReader(HLogSplitter.java:745) at org.apache.hadoop.hbase.regionserver.wal.HLogSplitter.splitLogFile(HLogSplitter.java:436) at org.apache.hadoop.hbase.regionserver.wal.HLogSplitter.splitLogFile(HLogSplitter.java:397) at org.apache.hadoop.hbase.regionserver.SplitLogWorker$1.exec(SplitLogWorker.java:111) at org.apache.hadoop.hbase.regionserver.SplitLogWorker.grabTask(SplitLogWorker.java:274) at org.apache.hadoop.hbase.regionserver.SplitLogWorker.taskLoop(SplitLogWorker.java:195) at org.apache.hadoop.hbase.regionserver.SplitLogWorker.run(SplitLogWorker.java:162) at java.lang.Thread.run(Thread.java:662) 2013-04-07 12:14:43,508 WARN [SplitLogWorker-asf001.sp2.ygridcore.net,60884,1365336878389] wal.HLogSplitter(444): Nothing to split in log file hdfs://localhost:60405/user/jenkins/hbase/.logs/asf001.sp2.ygridcore.net,35684,1365336878445-splitting/asf001.sp2.ygridcore.net%2C35684%2C1365336878445.1365336879032 2013-04-07 12:14:43,510 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(274): master:33796-0x13de46b5f8b0000 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/splitlog/.logs%2Fasf001.sp2.ygridcore.net%2C35684%2C1365336878445-splitting%2Fasf001.sp2.ygridcore.net%252C35684%252C1365336878445.1365336879032 2013-04-07 12:14:43,510 DEBUG [RegionServer:0;asf001.sp2.ygridcore.net,60884,1365336878389-EventThread] zookeeper.ZooKeeperWatcher(274): regionserver:60884-0x13de46b5f8b0001 Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/splitlog/.logs%2Fasf001.sp2.ygridcore.net%2C35684%2C1365336878445-splitting%2Fasf001.sp2.ygridcore.net%252C35684%252C1365336878445.1365336879032 2013-04-07 12:14:43,512 INFO [SplitLogWorker-asf001.sp2.ygridcore.net,60884,1365336878389] regionserver.SplitLogWorker(381): successfully transitioned task /hbase/splitlog/.logs%2Fasf001.sp2.ygridcore.net%2C35684%2C1365336878445-splitting%2Fasf001.sp2.ygridcore.net%252C35684%252C1365336878445.1365336879032 to final state DONE asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:14:43,512 INFO [SplitLogWorker-asf001.sp2.ygridcore.net,60884,1365336878389] regionserver.SplitLogWorker(315): worker asf001.sp2.ygridcore.net,60884,1365336878389 done with task /hbase/splitlog/.logs%2Fasf001.sp2.ygridcore.net%2C35684%2C1365336878445-splitting%2Fasf001.sp2.ygridcore.net%252C35684%252C1365336878445.1365336879032 in 511ms 2013-04-07 12:14:43,513 INFO [pool-1-thread-1-EventThread] master.SplitLogManager(512): task /hbase/splitlog/.logs%2Fasf001.sp2.ygridcore.net%2C35684%2C1365336878445-splitting%2Fasf001.sp2.ygridcore.net%252C35684%252C1365336878445.1365336879032 entered state: DONE asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:14:43,515 DEBUG [pool-1-thread-1] hbase.HRegionInfo(954): Current INFO from scan results = {NAME => 'test1365336881340,,1365336881424.c9e9e308a02801bb891879b6cf471381.', STARTKEY => '', ENDKEY => '', ENCODED => c9e9e308a02801bb891879b6cf471381,} 2013-04-07 12:14:43,518 DEBUG [pool-1-thread-1] zookeeper.ZKUtil(1680): hconnection-0xfe04c6-0x13de46b5f8b0007 Retrieved 6 byte(s) of data from znode /hbase/table/test1365336881340; data=PBUF\x08\x00 2013-04-07 12:14:43,519 DEBUG [pool-1-thread-1-EventThread] wal.HLogSplitter(600): Archived processed log hdfs://localhost:60405/user/jenkins/hbase/.logs/asf001.sp2.ygridcore.net,35684,1365336878445-splitting/asf001.sp2.ygridcore.net%2C35684%2C1365336878445.1365336879032 to hdfs://localhost:60405/user/jenkins/hbase/.oldlogs/asf001.sp2.ygridcore.net%2C35684%2C1365336878445.1365336879032 2013-04-07 12:14:43,520 DEBUG [pool-1-thread-1] zookeeper.ZKUtil(1680): hconnection-0xfe04c6-0x13de46b5f8b0007 Retrieved 43 byte(s) of data from znode /hbase/meta-region-server; data=asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:14:43,520 INFO [pool-1-thread-1-EventThread] master.SplitLogManager(425): Done splitting /hbase/splitlog/.logs%2Fasf001.sp2.ygridcore.net%2C35684%2C1365336878445-splitting%2Fasf001.sp2.ygridcore.net%252C35684%252C1365336878445.1365336879032 2013-04-07 12:14:43,522 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(274): master:33796-0x13de46b5f8b0000 Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/splitlog/.logs%2Fasf001.sp2.ygridcore.net%2C35684%2C1365336878445-splitting%2Fasf001.sp2.ygridcore.net%252C35684%252C1365336878445.1365336879032 2013-04-07 12:14:43,522 DEBUG [RegionServer:0;asf001.sp2.ygridcore.net,60884,1365336878389-EventThread] zookeeper.ZooKeeperWatcher(274): regionserver:60884-0x13de46b5f8b0001 Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/splitlog 2013-04-07 12:14:43,522 DEBUG [RegionServer:0;asf001.sp2.ygridcore.net,60884,1365336878389-EventThread] regionserver.SplitLogWorker(506): tasks arrived or departed 2013-04-07 12:14:43,523 DEBUG [pool-1-thread-1] zookeeper.ZKUtil(1680): hconnection-0xfe04c6-0x13de46b5f8b0007 Retrieved 43 byte(s) of data from znode /hbase/meta-region-server; data=asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:14:43,523 INFO [MASTER_META_SERVER_OPERATIONS-asf001.sp2.ygridcore.net,33796,1365336878036-0] master.SplitLogManager(318): finished splitting (more than or equal to) 0 bytes in 1 log files in [hdfs://localhost:60405/user/jenkins/hbase/.logs/asf001.sp2.ygridcore.net,35684,1365336878445-splitting] in 661ms 2013-04-07 12:14:43,523 DEBUG [pool-1-thread-1-EventThread] master.SplitLogManager$DeleteAsyncCallback(1191): deleted /hbase/splitlog/.logs%2Fasf001.sp2.ygridcore.net%2C35684%2C1365336878445-splitting%2Fasf001.sp2.ygridcore.net%252C35684%252C1365336878445.1365336879032 2013-04-07 12:14:43,525 DEBUG [pool-1-thread-1] zookeeper.ZKUtil(1680): hconnection-0xfe04c6-0x13de46b5f8b0007 Retrieved 43 byte(s) of data from znode /hbase/meta-region-server; data=asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:14:43,526 DEBUG [MASTER_META_SERVER_OPERATIONS-asf001.sp2.ygridcore.net,33796,1365336878036-0] zookeeper.ZKUtil(1680): master:33796-0x13de46b5f8b0000 Retrieved 43 byte(s) of data from znode /hbase/meta-region-server and set watcher; asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:14:43,528 DEBUG [MASTER_META_SERVER_OPERATIONS-asf001.sp2.ygridcore.net,33796,1365336878036-0] zookeeper.ZKUtil(1680): hconnection-0x1f327e-0x13de46b5f8b0003 Retrieved 43 byte(s) of data from znode /hbase/meta-region-server; data=asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:14:43,528 DEBUG [MASTER_META_SERVER_OPERATIONS-asf001.sp2.ygridcore.net,33796,1365336878036-0] client.ClientScanner(96): Creating scanner over .META. starting at key '' 2013-04-07 12:14:43,528 DEBUG [MASTER_META_SERVER_OPERATIONS-asf001.sp2.ygridcore.net,33796,1365336878036-0] client.ClientScanner(209): Advancing internal scanner to startKey at '' 2013-04-07 12:14:43,529 DEBUG [MASTER_META_SERVER_OPERATIONS-asf001.sp2.ygridcore.net,33796,1365336878036-0] zookeeper.ZKUtil(1680): hconnection-0x1f327e-0x13de46b5f8b0003 Retrieved 43 byte(s) of data from znode /hbase/meta-region-server; data=asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:14:43,534 DEBUG [MASTER_META_SERVER_OPERATIONS-asf001.sp2.ygridcore.net,33796,1365336878036-0] client.ClientScanner(196): Finished with scanning at {NAME => '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192,} 2013-04-07 12:14:43,534 DEBUG [pool-1-thread-1] client.MetaScanner(251): Current INFO from scan results = {NAME => 'test1365336881340,,1365336881424.c9e9e308a02801bb891879b6cf471381.', STARTKEY => '', ENDKEY => '', ENCODED => c9e9e308a02801bb891879b6cf471381,} 2013-04-07 12:14:43,534 INFO [MASTER_META_SERVER_OPERATIONS-asf001.sp2.ygridcore.net,33796,1365336878036-0] handler.ServerShutdownHandler(192): Reassigning 0 region(s) that asf001.sp2.ygridcore.net,35684,1365336878445 was carrying (and 0 regions(s) that were opening on this server) 2013-04-07 12:14:43,534 DEBUG [pool-1-thread-1] client.MetaScanner(198): Scanning .META. starting at row=test1365336881340,,00000000000000 for max=10 rows using hconnection-0xfe04c6 2013-04-07 12:14:43,534 INFO [MASTER_META_SERVER_OPERATIONS-asf001.sp2.ygridcore.net,33796,1365336878036-0] handler.ServerShutdownHandler(270): Finished processing of shutdown of asf001.sp2.ygridcore.net,35684,1365336878445 2013-04-07 12:14:43,536 DEBUG [pool-1-thread-1] zookeeper.ZKUtil(1680): hconnection-0xfe04c6-0x13de46b5f8b0007 Retrieved 43 byte(s) of data from znode /hbase/meta-region-server; data=asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:14:43,539 DEBUG [pool-1-thread-1] client.MetaScanner(251): Current INFO from scan results = {NAME => 'test1365336881340,,1365336881424.c9e9e308a02801bb891879b6cf471381.', STARTKEY => '', ENDKEY => '', ENCODED => c9e9e308a02801bb891879b6cf471381,} 2013-04-07 12:14:43,540 DEBUG [pool-1-thread-1] client.HConnectionManager$HConnectionImplementation(1433): Cached location for test1365336881340,,1365336881424.c9e9e308a02801bb891879b6cf471381. is asf001.sp2.ygridcore.net:60884 2013-04-07 12:14:43,542 INFO [pool-1-thread-1] hbase.TestZooKeeper(249): Putting table test1365336881340 2013-04-07 12:14:43,573 INFO [pool-1-thread-1] hbase.ResourceChecker(171): after: TestZooKeeper#testRegionServerSessionExpired Thread=193 (was 207), OpenFileDescriptor=312 (was 336), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=340 (was 340), ProcessCount=175 (was 175), AvailableMemoryMB=6903 (was 6903), ConnectionCount=3 (was 4) 2013-04-07 12:14:43,593 INFO [pool-1-thread-1] hbase.ResourceChecker(147): before: TestZooKeeper#testGetChildDataAndWatchForNewChildrenShouldNotThrowNPE Thread=193, OpenFileDescriptor=315, MaxFileDescriptor=60000, SystemLoadAverage=340, ProcessCount=176, AvailableMemoryMB=6903, ConnectionCount=3 2013-04-07 12:14:43,594 DEBUG [pool-1-thread-1] zookeeper.ZKConfig(84): Skipped reading ZK properties file 'zoo.cfg' since 'hbase.config.read.zookeeper.config' was not set to true 2013-04-07 12:14:43,595 DEBUG [pool-1-thread-1] zookeeper.ZKUtil(119): testGetChildDataAndWatchForNewChildrenShouldNotThrowNPE opening connection to ZooKeeper with ensemble (localhost:51597) 2013-04-07 12:14:43,595 INFO [pool-1-thread-1] zookeeper.RecoverableZooKeeper(119): The identifier of this process is testGetChildDataAndWatchForNewChildrenShouldNotThrowNPE 2013-04-07 12:14:43,598 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(274): testGetChildDataAndWatchForNewChildrenShouldNotThrowNPE Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2013-04-07 12:14:43,599 DEBUG [pool-1-thread-1] zookeeper.ZKUtil(513): testGetChildDataAndWatchForNewChildrenShouldNotThrowNPE Unable to list children of znode /wrongNode because node does not exist (not an error) 2013-04-07 12:14:43,604 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(351): testGetChildDataAndWatchForNewChildrenShouldNotThrowNPE-0x13de46b5f8b000c connected 2013-04-07 12:14:43,618 INFO [pool-1-thread-1] hbase.ResourceChecker(171): after: TestZooKeeper#testGetChildDataAndWatchForNewChildrenShouldNotThrowNPE Thread=195 (was 193) - Thread LEAK? -, OpenFileDescriptor=321 (was 315) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=340 (was 340), ProcessCount=175 (was 176), AvailableMemoryMB=6900 (was 6903), ConnectionCount=3 (was 3) 2013-04-07 12:16:05,149 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(274): testGetChildDataAndWatchForNewChildrenShouldNotThrowNPE-0x13de46b5f8b000c Received ZooKeeper Event, type=None, state=Disconnected, path=null 2013-04-07 12:16:05,149 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(356): testGetChildDataAndWatchForNewChildrenShouldNotThrowNPE-0x13de46b5f8b000c Received Disconnected from ZooKeeper, ignoring 2013-04-07 12:16:05,156 WARN [org.apache.hadoop.hdfs.server.datanode.DataXceiverServer@1ee04fd] datanode.DataXceiverServer(138): DatanodeRegistration(127.0.0.1:43960, storageID=DS-1272336944-67.195.138.31-43960-1365336877572, infoPort=43017, ipcPort=41501):DataXceiveServer:java.nio.channels.AsynchronousCloseException at java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:185) at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:159) at sun.nio.ch.ServerSocketAdaptor.accept(ServerSocketAdaptor.java:84) at org.apache.hadoop.hdfs.server.datanode.DataXceiverServer.run(DataXceiverServer.java:131) at java.lang.Thread.run(Thread.java:662) 2013-04-07 12:16:06,156 WARN [pool-1-thread-1] datanode.FSDatasetAsyncDiskService(121): AsyncDiskService has already shut down. Shutting down DataNode 0 2013-04-07 12:16:06,156 INFO [pool-1-thread-1] log.Slf4jLog(67): Stopped SelectChannelConnector@localhost:0 2013-04-07 12:16:06,158 WARN [org.apache.hadoop.hdfs.server.datanode.DataXceiverServer@1984a9d] datanode.DataXceiverServer(138): DatanodeRegistration(127.0.0.1:37034, storageID=DS-155880058-67.195.138.31-37034-1365336877245, infoPort=48590, ipcPort=59471):DataXceiveServer:java.nio.channels.AsynchronousCloseException at java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:185) at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:159) at sun.nio.ch.ServerSocketAdaptor.accept(ServerSocketAdaptor.java:84) at org.apache.hadoop.hdfs.server.datanode.DataXceiverServer.run(DataXceiverServer.java:131) at java.lang.Thread.run(Thread.java:662) 2013-04-07 12:16:07,159 WARN [pool-1-thread-1] datanode.FSDatasetAsyncDiskService(121): AsyncDiskService has already shut down. 2013-04-07 12:16:07,159 INFO [pool-1-thread-1] log.Slf4jLog(67): Stopped SelectChannelConnector@localhost:0 2013-04-07 12:16:07,261 WARN [org.apache.hadoop.hdfs.server.namenode.FSNamesystem$ReplicationMonitor@4c6320] namenode.FSNamesystem$ReplicationMonitor(2868): ReplicationMonitor thread received InterruptedException.java.lang.InterruptedException: sleep interrupted 2013-04-07 12:16:07,283 INFO [pool-1-thread-1] hbase.HBaseTestingUtility(897): Minicluster is down 2013-04-07 12:14:43,639 INFO [pool-1-thread-1] hbase.ResourceChecker(147): before: TestZooKeeper#testMultipleZK Thread=196, OpenFileDescriptor=323, MaxFileDescriptor=60000, SystemLoadAverage=340, ProcessCount=175, AvailableMemoryMB=6899, ConnectionCount=3 2013-04-07 12:14:43,641 DEBUG [pool-1-thread-1] zookeeper.ZKUtil(1680): hconnection-0xfe04c6-0x13de46b5f8b0007 Retrieved 43 byte(s) of data from znode /hbase/meta-region-server; data=asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:14:43,642 DEBUG [pool-1-thread-1] zookeeper.ZKConfig(84): Skipped reading ZK properties file 'zoo.cfg' since 'hbase.config.read.zookeeper.config' was not set to true 2013-04-07 12:14:43,643 DEBUG [pool-1-thread-1] zookeeper.ZKUtil(119): hconnection-0x15e7597 opening connection to ZooKeeper with ensemble (127.0.0.1:51597) 2013-04-07 12:14:43,644 INFO [pool-1-thread-1] zookeeper.RecoverableZooKeeper(119): The identifier of this process is hconnection-0x15e7597 2013-04-07 12:14:43,646 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(274): hconnection-0x15e7597 Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2013-04-07 12:14:43,647 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(351): hconnection-0x15e7597-0x13de46b5f8b000d connected 2013-04-07 12:14:43,648 DEBUG [pool-1-thread-1] zookeeper.ZKUtil(1680): hconnection-0x15e7597-0x13de46b5f8b000d Retrieved 42 byte(s) of data from znode /hbase/hbaseid; data=PBUF\x0A$cd034c7a-08a3-4d5e-8... 2013-04-07 12:14:43,648 INFO [pool-1-thread-1] client.HConnectionManager$HConnectionImplementation(699): ClusterId is cd034c7a-08a3-4d5e-8eea-f9d6aa6dd983 2013-04-07 12:14:43,649 DEBUG [pool-1-thread-1] zookeeper.ZKUtil(1680): hconnection-0x15e7597-0x13de46b5f8b000d Retrieved 43 byte(s) of data from znode /hbase/meta-region-server; data=asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:14:43,651 DEBUG [pool-1-thread-1] zookeeper.ZKUtil(1680): hconnection-0xfe04c6-0x13de46b5f8b0007 Retrieved 43 byte(s) of data from znode /hbase/meta-region-server; data=asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:14:43,654 DEBUG [pool-1-thread-1] zookeeper.ZKUtil(1680): hconnection-0x15e7597-0x13de46b5f8b000d Retrieved 43 byte(s) of data from znode /hbase/meta-region-server; data=asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:14:43,673 INFO [pool-1-thread-1] hbase.ResourceChecker(171): after: TestZooKeeper#testMultipleZK Thread=198 (was 196) - Thread LEAK? -, OpenFileDescriptor=332 (was 323) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=340 (was 340), ProcessCount=175 (was 175), AvailableMemoryMB=6900 (was 6899) - AvailableMemoryMB LEAK? -, ConnectionCount=4 (was 3) - ConnectionCount LEAK? - 2013-04-07 12:14:43,692 INFO [pool-1-thread-1] hbase.ResourceChecker(147): before: TestZooKeeper#testMasterZKSessionRecoveryFailure Thread=198, OpenFileDescriptor=334, MaxFileDescriptor=60000, SystemLoadAverage=340, ProcessCount=175, AvailableMemoryMB=6900, ConnectionCount=4 2013-04-07 12:14:43,693 FATAL [Thread-200] master.HMaster(1968): Master server abort: loaded coprocessors are: [org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint] 2013-04-07 12:14:43,694 INFO [Thread-200] master.HMaster(2052): Primary Master trying to recover from ZooKeeper session expiry. 2013-04-07 12:14:43,694 INFO [Thread-200] zookeeper.RecoverableZooKeeper(131): Closing dead ZooKeeper connection, session was: 0x13de46b5f8b0000 2013-04-07 12:14:43,696 DEBUG [RegionServer:0;asf001.sp2.ygridcore.net,60884,1365336878389-EventThread] zookeeper.ZooKeeperWatcher(274): regionserver:60884-0x13de46b5f8b0001 Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2013-04-07 12:14:43,697 INFO [Thread-200] zookeeper.RecoverableZooKeeper(136): Recreated a ZooKeeper, session is: 0x0 2013-04-07 12:14:43,697 DEBUG [RegionServer:0;asf001.sp2.ygridcore.net,60884,1365336878389-EventThread] zookeeper.ZKUtil(424): regionserver:60884-0x13de46b5f8b0001 /hbase/master does not exist. Watcher is set. 2013-04-07 12:14:43,699 DEBUG [Thread-200-EventThread] zookeeper.ZooKeeperWatcher(274): master:33796-0x13de46b5f8b0000 Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2013-04-07 12:14:43,700 DEBUG [Thread-200-EventThread] zookeeper.ZooKeeperWatcher(351): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e connected 2013-04-07 12:14:43,701 DEBUG [pool-40-thread-1] zookeeper.ZKUtil(422): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Set watcher on existing znode /hbase/running 2013-04-07 12:14:43,702 DEBUG [pool-40-thread-1] zookeeper.ZKUtil(1680): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Retrieved 34 byte(s) of data from znode /hbase/running and set watcher; PBUF\x0A\x1CSun Apr 07 12:14:... 2013-04-07 12:14:43,703 DEBUG [RegionServer:0;asf001.sp2.ygridcore.net,60884,1365336878389-EventThread] zookeeper.ZooKeeperWatcher(274): regionserver:60884-0x13de46b5f8b0001 Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2013-04-07 12:14:43,705 WARN [pool-40-thread-1] hbase.ZNodeClearer(58): No filename given to save the znode used, it won't be saved (Environment variable HBASE_ZNODE_FILE is not set). 2013-04-07 12:14:43,705 INFO [pool-40-thread-1] master.ActiveMasterManager(159): Master=asf001.sp2.ygridcore.net,33796,1365336878036 2013-04-07 12:14:43,705 DEBUG [RegionServer:0;asf001.sp2.ygridcore.net,60884,1365336878389-EventThread] zookeeper.ZKUtil(1680): regionserver:60884-0x13de46b5f8b0001 Retrieved 43 byte(s) of data from znode /hbase/master and set watcher; PBUF\x0A%\x0A\x18asf001.sp2.y... 2013-04-07 12:14:43,706 INFO [pool-40-thread-1] master.SplitLogManager(181): timeout = 120000 2013-04-07 12:14:43,706 INFO [pool-40-thread-1] master.SplitLogManager(182): unassigned timeout = 180000 2013-04-07 12:14:43,707 INFO [pool-40-thread-1] master.SplitLogManager(858): found 0 orphan tasks and 0 rescan nodes 2013-04-07 12:14:43,720 INFO [pool-40-thread-1] util.FSTableDescriptors(630): Current tableInfoPath = hdfs://localhost:60405/user/jenkins/hbase/.META./.tableinfo.0000000001 2013-04-07 12:14:43,724 INFO [pool-40-thread-1] util.FSTableDescriptors(634): TableInfo already exists.. Skipping creation 2013-04-07 12:14:43,730 DEBUG [pool-40-thread-1] fs.HFileSystem(199): Starting addLocationsOrderInterceptor with class class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2013-04-07 12:14:43,731 INFO [pool-40-thread-1] fs.HFileSystem(246): Added intercepting call to namenode#getBlockLocations 2013-04-07 12:14:43,734 DEBUG [pool-40-thread-1] catalog.CatalogTracker(192): Starting catalog tracker org.apache.hadoop.hbase.catalog.CatalogTracker@141cd01 2013-04-07 12:14:43,735 DEBUG [pool-40-thread-1] zookeeper.ZKUtil(422): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Set watcher on existing znode /hbase/meta-region-server 2013-04-07 12:14:43,736 DEBUG [pool-40-thread-1] zookeeper.ZKUtil(1680): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Retrieved 43 byte(s) of data from znode /hbase/meta-region-server and set watcher; asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:14:43,737 DEBUG [pool-40-thread-1] zookeeper.ZKUtil(424): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e /hbase/balancer does not exist. Watcher is set. 2013-04-07 12:14:43,739 DEBUG [pool-40-thread-1] zookeeper.ZKUtil(1680): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Retrieved 6 byte(s) of data from znode /hbase/table/.META.; data=PBUF\x08\x00 2013-04-07 12:14:43,740 DEBUG [pool-40-thread-1] zookeeper.ZKUtil(1680): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Retrieved 6 byte(s) of data from znode /hbase/table/test1365336881340; data=PBUF\x08\x00 2013-04-07 12:14:43,742 DEBUG [pool-40-thread-1] zookeeper.ZKUtil(422): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Set watcher on existing znode /hbase/rs/asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:14:43,743 INFO [pool-40-thread-1] master.HMaster(615): Server active/primary master; asf001.sp2.ygridcore.net,33796,1365336878036, sessionid=0x13de46b5f8b000e, cluster-up flag was=true 2013-04-07 12:14:43,745 INFO [pool-40-thread-1] zookeeper.RecoverableZooKeeper(460): Node /hbase/online-snapshot/acquired already exists and this is not a retry 2013-04-07 12:14:43,748 INFO [pool-40-thread-1] procedure.ZKProcedureUtil(266): Clearing all procedure znodes: /hbase/online-snapshot/acquired /hbase/online-snapshot/reached /hbase/online-snapshot/abort 2013-04-07 12:14:43,751 DEBUG [pool-40-thread-1] procedure.ZKProcedureCoordinatorRpcs(194): Starting the controller for procedure member:asf001.sp2.ygridcore.net,33796,1365336878036 2013-04-07 12:14:43,752 WARN [pool-40-thread-1] snapshot.SnapshotManager(242): Couldn't delete working snapshot directory: hdfs://localhost:60405/user/jenkins/hbase/.snapshot/.tmp 2013-04-07 12:14:43,753 INFO [pool-40-thread-1] master.ServerManager(778): Waiting for region servers count to settle; currently checked in 1, slept for 0 ms, expecting minimum of 1, maximum of 2, timeout of 4500 ms, interval of 1500 ms. 2013-04-07 12:14:44,425 WARN [RegionServer:1;asf001.sp2.ygridcore.net,35684,1365336878445] zookeeper.RecoverableZooKeeper(236): Possibly transient ZooKeeper, quorum=localhost:51597, exception=org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /hbase/rs/asf001.sp2.ygridcore.net,35684,1365336878445 2013-04-07 12:14:44,425 INFO [RegionServer:1;asf001.sp2.ygridcore.net,35684,1365336878445] util.RetryCounter(54): Sleeping 4000ms before retry #2... 2013-04-07 12:14:45,255 INFO [pool-40-thread-1] master.ServerManager(778): Waiting for region servers count to settle; currently checked in 1, slept for 1502 ms, expecting minimum of 1, maximum of 2, timeout of 4500 ms, interval of 1500 ms. 2013-04-07 12:14:46,757 INFO [pool-40-thread-1] master.ServerManager(778): Waiting for region servers count to settle; currently checked in 1, slept for 3004 ms, expecting minimum of 1, maximum of 2, timeout of 4500 ms, interval of 1500 ms. 2013-04-07 12:14:48,260 INFO [pool-40-thread-1] master.ServerManager(795): Finished waiting for region servers count to settle; checked in 1, slept for 4507 ms, expecting minimum of 1, maximum of 2, master is running. 2013-04-07 12:14:48,263 INFO [pool-40-thread-1] master.MasterFileSystem(252): Log folder hdfs://localhost:60405/user/jenkins/hbase/.logs/asf001.sp2.ygridcore.net,60884,1365336878389 belongs to an existing region server 2013-04-07 12:14:48,264 INFO [pool-40-thread-1] master.MasterFileSystem(346): No logs to split 2013-04-07 12:14:48,264 INFO [pool-40-thread-1] master.MasterFileSystem(346): No logs to split 2013-04-07 12:14:48,267 DEBUG [pool-40-thread-1] zookeeper.ZKUtil(1680): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Retrieved 43 byte(s) of data from znode /hbase/meta-region-server and set watcher; asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:14:48,268 DEBUG [pool-40-thread-1] zookeeper.ZKUtil(1680): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Retrieved 43 byte(s) of data from znode /hbase/meta-region-server and set watcher; asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:14:48,270 DEBUG [pool-40-thread-1] zookeeper.ZKUtil(1680): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Retrieved 43 byte(s) of data from znode /hbase/meta-region-server and set watcher; asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:14:48,270 DEBUG [pool-40-thread-1] master.RegionStates(290): Online a region with current state=OFFLINE, expected state=OPEN, assigned to server: null expected asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:14:48,270 INFO [pool-40-thread-1] master.RegionStates(264): Region {NAME => '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192,} transitioned from {.META.,,1.1028785192 state=OFFLINE, ts=1365336888264, server=null} to {.META.,,1.1028785192 state=OPEN, ts=1365336888270, server=asf001.sp2.ygridcore.net,60884,1365336878389} 2013-04-07 12:14:48,270 INFO [pool-40-thread-1] master.RegionStates(299): Onlined region {NAME => '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192,} on asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:14:48,271 DEBUG [pool-40-thread-1] zookeeper.ZKUtil(1680): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Retrieved 43 byte(s) of data from znode /hbase/meta-region-server and set watcher; asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:14:48,272 INFO [pool-40-thread-1] master.HMaster(897): .META. assigned=0, rit=false, location=asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:14:48,272 INFO [pool-40-thread-1] master.ServerManager(533): AssignmentManager hasn't finished failover cleanup 2013-04-07 12:14:48,273 DEBUG [pool-40-thread-1] zookeeper.ZKUtil(1680): hconnection-0x1f327e-0x13de46b5f8b0003 Retrieved 43 byte(s) of data from znode /hbase/meta-region-server; data=asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:14:48,273 DEBUG [pool-40-thread-1] client.ClientScanner(96): Creating scanner over .META. starting at key '' 2013-04-07 12:14:48,273 DEBUG [pool-40-thread-1] client.ClientScanner(209): Advancing internal scanner to startKey at '' 2013-04-07 12:14:48,274 DEBUG [pool-40-thread-1] zookeeper.ZKUtil(1680): hconnection-0x1f327e-0x13de46b5f8b0003 Retrieved 43 byte(s) of data from znode /hbase/meta-region-server; data=asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:14:48,278 DEBUG [pool-40-thread-1] client.ClientScanner(196): Finished with scanning at {NAME => '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192,} 2013-04-07 12:14:48,279 INFO [pool-40-thread-1] catalog.MetaMigrationConvertingToPB(132): META already up-to date with PB serialization 2013-04-07 12:14:48,282 DEBUG [pool-40-thread-1] zookeeper.ZKUtil(1680): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Retrieved 6 byte(s) of data from znode /hbase/table/.META.; data=PBUF\x08\x00 2013-04-07 12:14:48,283 DEBUG [pool-40-thread-1] zookeeper.ZKUtil(1680): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Retrieved 6 byte(s) of data from znode /hbase/table/test1365336881340; data=PBUF\x08\x00 2013-04-07 12:14:48,285 DEBUG [pool-40-thread-1] zookeeper.ZKUtil(1680): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Retrieved 6 byte(s) of data from znode /hbase/table/.META.; data=PBUF\x08\x00 2013-04-07 12:14:48,286 DEBUG [pool-40-thread-1] zookeeper.ZKUtil(1680): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Retrieved 6 byte(s) of data from znode /hbase/table/test1365336881340; data=PBUF\x08\x00 2013-04-07 12:14:48,288 DEBUG [pool-40-thread-1] zookeeper.ZKUtil(1680): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Retrieved 6 byte(s) of data from znode /hbase/table/.META.; data=PBUF\x08\x00 2013-04-07 12:14:48,289 DEBUG [pool-40-thread-1] zookeeper.ZKUtil(1680): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Retrieved 6 byte(s) of data from znode /hbase/table/test1365336881340; data=PBUF\x08\x00 2013-04-07 12:14:48,290 DEBUG [pool-40-thread-1] zookeeper.ZKUtil(1680): hconnection-0x1f327e-0x13de46b5f8b0003 Retrieved 43 byte(s) of data from znode /hbase/meta-region-server; data=asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:14:48,291 DEBUG [pool-40-thread-1] client.ClientScanner(96): Creating scanner over .META. starting at key '' 2013-04-07 12:14:48,291 DEBUG [pool-40-thread-1] client.ClientScanner(209): Advancing internal scanner to startKey at '' 2013-04-07 12:14:48,292 DEBUG [pool-40-thread-1] zookeeper.ZKUtil(1680): hconnection-0x1f327e-0x13de46b5f8b0003 Retrieved 43 byte(s) of data from znode /hbase/meta-region-server; data=asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:14:48,296 DEBUG [pool-40-thread-1] client.ClientScanner(196): Finished with scanning at {NAME => '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192,} 2013-04-07 12:14:48,296 DEBUG [pool-40-thread-1] master.RegionStates(290): Online a region with current state=OFFLINE, expected state=OPEN, assigned to server: null expected asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:14:48,297 INFO [pool-40-thread-1] master.RegionStates(264): Region {NAME => 'test1365336881340,,1365336881424.c9e9e308a02801bb891879b6cf471381.', STARTKEY => '', ENDKEY => '', ENCODED => c9e9e308a02801bb891879b6cf471381,} transitioned from {test1365336881340,,1365336881424.c9e9e308a02801bb891879b6cf471381. state=OFFLINE, ts=1365336888296, server=null} to {test1365336881340,,1365336881424.c9e9e308a02801bb891879b6cf471381. state=OPEN, ts=1365336888297, server=asf001.sp2.ygridcore.net,60884,1365336878389} 2013-04-07 12:14:48,297 INFO [pool-40-thread-1] master.RegionStates(299): Onlined region {NAME => 'test1365336881340,,1365336881424.c9e9e308a02801bb891879b6cf471381.', STARTKEY => '', ENDKEY => '', ENCODED => c9e9e308a02801bb891879b6cf471381,} on asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:14:48,298 INFO [pool-40-thread-1] master.AssignmentManager(439): Found regions out on cluster or in RIT; failover 2013-04-07 12:14:48,301 DEBUG [pool-40-thread-1] zookeeper.ZKUtil(1680): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Retrieved 6 byte(s) of data from znode /hbase/table/.META.; data=PBUF\x08\x00 2013-04-07 12:14:48,302 DEBUG [pool-40-thread-1] zookeeper.ZKUtil(1680): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Retrieved 6 byte(s) of data from znode /hbase/table/test1365336881340; data=PBUF\x08\x00 2013-04-07 12:14:48,304 DEBUG [pool-40-thread-1] zookeeper.ZKUtil(1680): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Retrieved 6 byte(s) of data from znode /hbase/table/.META.; data=PBUF\x08\x00 2013-04-07 12:14:48,305 DEBUG [pool-40-thread-1] zookeeper.ZKUtil(1680): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Retrieved 6 byte(s) of data from znode /hbase/table/test1365336881340; data=PBUF\x08\x00 2013-04-07 12:14:48,306 INFO [pool-40-thread-1] master.HMaster(792): Master has completed initialization 2013-04-07 12:14:48,306 INFO [Thread-200] hbase.TestZooKeeper(241): Creating table test1365336888306 2013-04-07 12:14:48,312 DEBUG [IPC Server handler 1 on 33796] zookeeper.ZKUtil(1680): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Retrieved 43 byte(s) of data from znode /hbase/meta-region-server and set watcher; asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:14:48,312 DEBUG [IPC Server handler 1 on 33796] master.TableLockManager$ZKTableLockManager$TableLockImpl(225): Attempt to acquire table write lock on :test1365336888306 for:C_M_CREATE_TABLE 2013-04-07 12:14:48,319 DEBUG [IPC Server handler 1 on 33796] zookeeper.ZKUtil(1680): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Retrieved 84 byte(s) of data from znode /hbase/table-lock/test1365336888306/write-master:337960000000000; data=PBUF\x0A\x11test1365336888306... 2013-04-07 12:14:48,320 DEBUG [IPC Server handler 1 on 33796] lock.ZKInterProcessLockBase(220): Successfully acquired a lock for /hbase/table-lock/test1365336888306/write-master:337960000000000 2013-04-07 12:14:48,320 DEBUG [IPC Server handler 1 on 33796] master.TableLockManager$ZKTableLockManager$TableLockImpl(246): Acquired table write lock on :test1365336888306 for:C_M_CREATE_TABLE 2013-04-07 12:14:48,321 DEBUG [IPC Server handler 1 on 33796] zookeeper.ZKUtil(1680): hconnection-0x1f327e-0x13de46b5f8b0003 Retrieved 43 byte(s) of data from znode /hbase/meta-region-server; data=asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:14:48,321 DEBUG [IPC Server handler 1 on 33796] client.ClientScanner(96): Creating scanner over .META. starting at key 'test1365336888306,,' 2013-04-07 12:14:48,321 DEBUG [IPC Server handler 1 on 33796] client.ClientScanner(209): Advancing internal scanner to startKey at 'test1365336888306,,' 2013-04-07 12:14:48,322 DEBUG [IPC Server handler 1 on 33796] zookeeper.ZKUtil(1680): hconnection-0x1f327e-0x13de46b5f8b0003 Retrieved 43 byte(s) of data from znode /hbase/meta-region-server; data=asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:14:48,325 DEBUG [IPC Server handler 1 on 33796] client.ClientScanner(196): Finished with scanning at {NAME => '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192,} 2013-04-07 12:14:48,330 INFO [MASTER_TABLE_OPERATIONS-asf001.sp2.ygridcore.net,33796,1365336878036-0] handler.CreateTableHandler(146): Attempting to create the table test1365336888306 2013-04-07 12:14:48,331 DEBUG [Thread-200] client.MetaScanner(198): Scanning .META. starting at row=test1365336888306,,00000000000000 for max=2147483647 rows using hconnection-0xfe04c6 2013-04-07 12:14:48,332 DEBUG [Thread-200] zookeeper.ZKUtil(1680): hconnection-0xfe04c6-0x13de46b5f8b0007 Retrieved 43 byte(s) of data from znode /hbase/meta-region-server; data=asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:14:48,349 INFO [RegionOpenAndInitThread-test1365336888306-1] regionserver.HRegion(3862): creating HRegion test1365336888306 HTD == 'test1365336888306', {NAME => 'fam', DATA_BLOCK_ENCODING => 'NONE', BLOOMFILTER => 'NONE', REPLICATION_SCOPE => '0', VERSIONS => '3', COMPRESSION => 'NONE', MIN_VERSIONS => '0', TTL => '2147483647', KEEP_DELETED_CELLS => 'false', BLOCKSIZE => '65536', IN_MEMORY => 'false', ENCODE_ON_DISK => 'true', BLOCKCACHE => 'true'} RootDir = hdfs://localhost:60405/user/jenkins/hbase/.tmp Table name == test1365336888306 2013-04-07 12:14:48,352 DEBUG [RegionOpenAndInitThread-test1365336888306-1] util.FSUtils(211): Creating file=hdfs://localhost:60405/user/jenkins/hbase/.tmp/test1365336888306/da419688e5c79938e207c0d3c056a9ed/.regioninfo with permission=rwxrwxrwx 2013-04-07 12:14:48,365 DEBUG [RegionOpenAndInitThread-test1365336888306-1] regionserver.HRegion(496): Instantiated test1365336888306,,1365336888309.da419688e5c79938e207c0d3c056a9ed. 2013-04-07 12:14:48,365 DEBUG [RegionOpenAndInitThread-test1365336888306-1] regionserver.HRegion(889): Closing test1365336888306,,1365336888309.da419688e5c79938e207c0d3c056a9ed.: disabling compactions & flushes 2013-04-07 12:14:48,365 DEBUG [RegionOpenAndInitThread-test1365336888306-1] regionserver.HRegion(911): Updates disabled for region test1365336888306,,1365336888309.da419688e5c79938e207c0d3c056a9ed. 2013-04-07 12:14:48,366 INFO [RegionOpenAndInitThread-test1365336888306-1] regionserver.HRegion(968): Closed test1365336888306,,1365336888309.da419688e5c79938e207c0d3c056a9ed. 2013-04-07 12:14:48,369 DEBUG [MASTER_TABLE_OPERATIONS-asf001.sp2.ygridcore.net,33796,1365336878036-0] zookeeper.ZKUtil(1680): hconnection-0x1f327e-0x13de46b5f8b0003 Retrieved 43 byte(s) of data from znode /hbase/meta-region-server; data=asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:14:48,370 DEBUG [MASTER_TABLE_OPERATIONS-asf001.sp2.ygridcore.net,33796,1365336878036-0] zookeeper.ZKUtil(1680): hconnection-0x1f327e-0x13de46b5f8b0003 Retrieved 43 byte(s) of data from znode /hbase/meta-region-server; data=asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:14:48,373 INFO [MASTER_TABLE_OPERATIONS-asf001.sp2.ygridcore.net,33796,1365336878036-0] catalog.MetaEditor(254): Added 1 regions in META 2013-04-07 12:14:48,374 INFO [MASTER_TABLE_OPERATIONS-asf001.sp2.ygridcore.net,33796,1365336878036-0] master.AssignmentManager(2363): Not use bulk assigning since we are assigning only 1 region(s) to 1 server(s) 2013-04-07 12:14:48,374 DEBUG [MASTER_TABLE_OPERATIONS-asf001.sp2.ygridcore.net,33796,1365336878036-0] master.AssignmentManager(1444): Bulk assigning 1 region(s) to asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:14:48,374 INFO [MASTER_TABLE_OPERATIONS-asf001.sp2.ygridcore.net,33796,1365336878036-0] master.RegionStates(264): Region {NAME => 'test1365336888306,,1365336888309.da419688e5c79938e207c0d3c056a9ed.', STARTKEY => '', ENDKEY => '', ENCODED => da419688e5c79938e207c0d3c056a9ed,} transitioned from {test1365336888306,,1365336888309.da419688e5c79938e207c0d3c056a9ed. state=OFFLINE, ts=1365336888374, server=null} to {test1365336888306,,1365336888309.da419688e5c79938e207c0d3c056a9ed. state=OFFLINE, ts=1365336888374, server=null} 2013-04-07 12:14:48,374 DEBUG [MASTER_TABLE_OPERATIONS-asf001.sp2.ygridcore.net,33796,1365336878036-0] zookeeper.ZKAssign(177): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Async create of unassigned node for da419688e5c79938e207c0d3c056a9ed with OFFLINE state 2013-04-07 12:14:48,387 DEBUG [Thread-200-EventThread] zookeeper.ZooKeeperWatcher(274): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/region-in-transition 2013-04-07 12:14:48,388 DEBUG [Thread-200-EventThread] master.OfflineCallback(69): rs={test1365336888306,,1365336888309.da419688e5c79938e207c0d3c056a9ed. state=OFFLINE, ts=1365336888374, server=null}, server=asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:14:48,390 DEBUG [Thread-200-EventThread] master.OfflineCallback$ExistCallback(106): rs={test1365336888306,,1365336888309.da419688e5c79938e207c0d3c056a9ed. state=OFFLINE, ts=1365336888374, server=null}, server=asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:14:48,390 DEBUG [hbase-am-zkevent-worker-pool-2-thread-17] zookeeper.ZKUtil(1680): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Retrieved 120 byte(s) of data from znode /hbase/region-in-transition/da419688e5c79938e207c0d3c056a9ed and set watcher; region=test1365336888306,,1365336888309.da419688e5c79938e207c0d3c056a9ed., state=M_ZK_REGION_OFFLINE, servername=asf001.sp2.ygridcore.net,60884,1365336878389, createTime=1365336888374, payload.length=0 2013-04-07 12:14:48,395 INFO [MASTER_TABLE_OPERATIONS-asf001.sp2.ygridcore.net,33796,1365336878036-0] master.AssignmentManager(1482): asf001.sp2.ygridcore.net,60884,1365336878389 unassigned znodes=1 of total=1 2013-04-07 12:14:48,395 INFO [MASTER_TABLE_OPERATIONS-asf001.sp2.ygridcore.net,33796,1365336878036-0] master.RegionStates(264): Region {NAME => 'test1365336888306,,1365336888309.da419688e5c79938e207c0d3c056a9ed.', STARTKEY => '', ENDKEY => '', ENCODED => da419688e5c79938e207c0d3c056a9ed,} transitioned from {test1365336888306,,1365336888309.da419688e5c79938e207c0d3c056a9ed. state=OFFLINE, ts=1365336888374, server=null} to {test1365336888306,,1365336888309.da419688e5c79938e207c0d3c056a9ed. state=PENDING_OPEN, ts=1365336888395, server=asf001.sp2.ygridcore.net,60884,1365336878389} 2013-04-07 12:14:48,396 INFO [PRI IPC Server handler 2 on 60884] regionserver.HRegionServer(3346): Received request to open region: test1365336888306,,1365336888309.da419688e5c79938e207c0d3c056a9ed. on asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:14:48,400 DEBUG [RS_OPEN_REGION-asf001.sp2.ygridcore.net,60884,1365336878389-1] zookeeper.ZKAssign(721): regionserver:60884-0x13de46b5f8b0001 Attempting to transition node da419688e5c79938e207c0d3c056a9ed from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING 2013-04-07 12:14:48,400 DEBUG [MASTER_TABLE_OPERATIONS-asf001.sp2.ygridcore.net,33796,1365336878036-0] master.AssignmentManager(1600): Bulk assigning done for asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:14:48,403 DEBUG [RS_OPEN_REGION-asf001.sp2.ygridcore.net,60884,1365336878389-1] zookeeper.ZKUtil(1680): regionserver:60884-0x13de46b5f8b0001 Retrieved 120 byte(s) of data from znode /hbase/region-in-transition/da419688e5c79938e207c0d3c056a9ed; data=region=test1365336888306,,1365336888309.da419688e5c79938e207c0d3c056a9ed., state=M_ZK_REGION_OFFLINE, servername=asf001.sp2.ygridcore.net,60884,1365336878389, createTime=1365336888374, payload.length=0 2013-04-07 12:14:48,404 DEBUG [MASTER_TABLE_OPERATIONS-asf001.sp2.ygridcore.net,33796,1365336878036-0] master.TableLockManager$ZKTableLockManager$TableLockImpl(253): Attempt to release table write lock on :test1365336888306 2013-04-07 12:14:48,405 DEBUG [Thread-200-EventThread] zookeeper.ZooKeeperWatcher(274): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/region-in-transition/da419688e5c79938e207c0d3c056a9ed 2013-04-07 12:14:48,405 DEBUG [RS_OPEN_REGION-asf001.sp2.ygridcore.net,60884,1365336878389-1] zookeeper.ZKAssign(784): regionserver:60884-0x13de46b5f8b0001 Successfully transitioned node da419688e5c79938e207c0d3c056a9ed from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING 2013-04-07 12:14:48,406 INFO [RS_OPEN_REGION-asf001.sp2.ygridcore.net,60884,1365336878389-1] regionserver.HRegion(4023): HRegion.openHRegion Region name ==test1365336888306,,1365336888309.da419688e5c79938e207c0d3c056a9ed. 2013-04-07 12:14:48,406 DEBUG [RS_OPEN_REGION-asf001.sp2.ygridcore.net,60884,1365336878389-1] regionserver.HRegion(4025): Opening region: {NAME => 'test1365336888306,,1365336888309.da419688e5c79938e207c0d3c056a9ed.', STARTKEY => '', ENDKEY => '', ENCODED => da419688e5c79938e207c0d3c056a9ed,} 2013-04-07 12:14:48,408 DEBUG [RS_OPEN_REGION-asf001.sp2.ygridcore.net,60884,1365336878389-1] regionserver.MetricsRegionSourceImpl(54): Creating new MetricsRegionSourceImpl for table test1365336888306 da419688e5c79938e207c0d3c056a9ed 2013-04-07 12:14:48,408 DEBUG [RS_OPEN_REGION-asf001.sp2.ygridcore.net,60884,1365336878389-1] regionserver.HRegion(496): Instantiated test1365336888306,,1365336888309.da419688e5c79938e207c0d3c056a9ed. 2013-04-07 12:14:48,408 DEBUG [MASTER_TABLE_OPERATIONS-asf001.sp2.ygridcore.net,33796,1365336878036-0] lock.ZKInterProcessLockBase(297): Successfully released /hbase/table-lock/test1365336888306/write-master:337960000000000 2013-04-07 12:14:48,408 DEBUG [MASTER_TABLE_OPERATIONS-asf001.sp2.ygridcore.net,33796,1365336878036-0] master.TableLockManager$ZKTableLockManager$TableLockImpl(269): Released table lock on :test1365336888306 2013-04-07 12:14:48,409 DEBUG [hbase-am-zkevent-worker-pool-2-thread-18] zookeeper.ZKUtil(1680): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Retrieved 120 byte(s) of data from znode /hbase/region-in-transition/da419688e5c79938e207c0d3c056a9ed and set watcher; region=test1365336888306,,1365336888309.da419688e5c79938e207c0d3c056a9ed., state=RS_ZK_REGION_OPENING, servername=asf001.sp2.ygridcore.net,60884,1365336878389, createTime=1365336888403, payload.length=0 2013-04-07 12:14:48,409 DEBUG [hbase-am-zkevent-worker-pool-20-thread-2] zookeeper.ZKUtil(1680): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Retrieved 120 byte(s) of data from znode /hbase/region-in-transition/da419688e5c79938e207c0d3c056a9ed and set watcher; region=test1365336888306,,1365336888309.da419688e5c79938e207c0d3c056a9ed., state=RS_ZK_REGION_OPENING, servername=asf001.sp2.ygridcore.net,60884,1365336878389, createTime=1365336888403, payload.length=0 2013-04-07 12:14:48,409 DEBUG [hbase-am-zkevent-worker-pool-2-thread-18] master.AssignmentManager(740): Handling transition=RS_ZK_REGION_OPENING, server=asf001.sp2.ygridcore.net,60884,1365336878389, region=da419688e5c79938e207c0d3c056a9ed, current state from region state map =null 2013-04-07 12:14:48,410 DEBUG [hbase-am-zkevent-worker-pool-20-thread-2] master.AssignmentManager(740): Handling transition=RS_ZK_REGION_OPENING, server=asf001.sp2.ygridcore.net,60884,1365336878389, region=da419688e5c79938e207c0d3c056a9ed, current state from region state map ={test1365336888306,,1365336888309.da419688e5c79938e207c0d3c056a9ed. state=PENDING_OPEN, ts=1365336888395, server=asf001.sp2.ygridcore.net,60884,1365336878389} 2013-04-07 12:14:48,410 INFO [hbase-am-zkevent-worker-pool-20-thread-2] master.RegionStates(264): Region {NAME => 'test1365336888306,,1365336888309.da419688e5c79938e207c0d3c056a9ed.', STARTKEY => '', ENDKEY => '', ENCODED => da419688e5c79938e207c0d3c056a9ed,} transitioned from {test1365336888306,,1365336888309.da419688e5c79938e207c0d3c056a9ed. state=PENDING_OPEN, ts=1365336888395, server=asf001.sp2.ygridcore.net,60884,1365336878389} to {test1365336888306,,1365336888309.da419688e5c79938e207c0d3c056a9ed. state=OPENING, ts=1365336888410, server=asf001.sp2.ygridcore.net,60884,1365336878389} 2013-04-07 12:14:48,411 DEBUG [hbase-am-zkevent-worker-pool-2-thread-18] zookeeper.ZKUtil(1680): hconnection-0x1f327e-0x13de46b5f8b0003 Retrieved 43 byte(s) of data from znode /hbase/meta-region-server; data=asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:14:48,413 DEBUG [hbase-am-zkevent-worker-pool-2-thread-18] zookeeper.ZKUtil(1680): hconnection-0x1f327e-0x13de46b5f8b0003 Retrieved 43 byte(s) of data from znode /hbase/meta-region-server; data=asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:14:48,415 INFO [hbase-am-zkevent-worker-pool-2-thread-18] master.RegionStates(264): Region {NAME => 'test1365336888306,,1365336888309.da419688e5c79938e207c0d3c056a9ed.', STARTKEY => '', ENDKEY => '', ENCODED => da419688e5c79938e207c0d3c056a9ed,} transitioned from {test1365336888306,,1365336888309.da419688e5c79938e207c0d3c056a9ed. state=OFFLINE, ts=1365336888415, server=null} to {test1365336888306,,1365336888309.da419688e5c79938e207c0d3c056a9ed. state=OPENING, ts=1365336888415, server=asf001.sp2.ygridcore.net,60884,1365336878389} 2013-04-07 12:14:48,416 INFO [StoreOpenerThread-test1365336888306,,1365336888309.da419688e5c79938e207c0d3c056a9ed.-1] compactions.CompactionConfiguration(85): Compaction configuration size [134217728, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; delete expired; major period 86400000, major jitter 0.200000 2013-04-07 12:14:48,419 INFO [RS_OPEN_REGION-asf001.sp2.ygridcore.net,60884,1365336878389-1] regionserver.HRegion(586): Onlined test1365336888306,,1365336888309.da419688e5c79938e207c0d3c056a9ed.; next sequenceid=1 2013-04-07 12:14:48,420 DEBUG [RS_OPEN_REGION-asf001.sp2.ygridcore.net,60884,1365336878389-1] zookeeper.ZKAssign(721): regionserver:60884-0x13de46b5f8b0001 Attempting to transition node da419688e5c79938e207c0d3c056a9ed from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENING 2013-04-07 12:14:48,421 DEBUG [RS_OPEN_REGION-asf001.sp2.ygridcore.net,60884,1365336878389-1] zookeeper.ZKUtil(1680): regionserver:60884-0x13de46b5f8b0001 Retrieved 120 byte(s) of data from znode /hbase/region-in-transition/da419688e5c79938e207c0d3c056a9ed; data=region=test1365336888306,,1365336888309.da419688e5c79938e207c0d3c056a9ed., state=RS_ZK_REGION_OPENING, servername=asf001.sp2.ygridcore.net,60884,1365336878389, createTime=1365336888403, payload.length=0 2013-04-07 12:14:48,424 DEBUG [Thread-200-EventThread] zookeeper.ZooKeeperWatcher(274): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/region-in-transition/da419688e5c79938e207c0d3c056a9ed 2013-04-07 12:14:48,424 DEBUG [RS_OPEN_REGION-asf001.sp2.ygridcore.net,60884,1365336878389-1] zookeeper.ZKAssign(784): regionserver:60884-0x13de46b5f8b0001 Successfully transitioned node da419688e5c79938e207c0d3c056a9ed from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENING 2013-04-07 12:14:48,425 INFO [PostOpenDeployTasks:da419688e5c79938e207c0d3c056a9ed] regionserver.HRegionServer(1563): Post open deploy tasks for region=test1365336888306,,1365336888309.da419688e5c79938e207c0d3c056a9ed. 2013-04-07 12:14:48,425 WARN [RegionServer:1;asf001.sp2.ygridcore.net,35684,1365336878445] zookeeper.RecoverableZooKeeper(236): Possibly transient ZooKeeper, quorum=localhost:51597, exception=org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /hbase/rs/asf001.sp2.ygridcore.net,35684,1365336878445 2013-04-07 12:14:48,426 INFO [RegionServer:1;asf001.sp2.ygridcore.net,35684,1365336878445] util.RetryCounter(54): Sleeping 8000ms before retry #3... 2013-04-07 12:14:48,426 DEBUG [PostOpenDeployTasks:da419688e5c79938e207c0d3c056a9ed] zookeeper.ZKUtil(1680): hconnection-0x43ad4a-0x13de46b5f8b0005 Retrieved 43 byte(s) of data from znode /hbase/meta-region-server; data=asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:14:48,427 DEBUG [hbase-am-zkevent-worker-pool-20-thread-3] zookeeper.ZKUtil(1680): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Retrieved 120 byte(s) of data from znode /hbase/region-in-transition/da419688e5c79938e207c0d3c056a9ed and set watcher; region=test1365336888306,,1365336888309.da419688e5c79938e207c0d3c056a9ed., state=RS_ZK_REGION_OPENING, servername=asf001.sp2.ygridcore.net,60884,1365336878389, createTime=1365336888421, payload.length=0 2013-04-07 12:14:48,428 DEBUG [hbase-am-zkevent-worker-pool-20-thread-3] master.AssignmentManager(740): Handling transition=RS_ZK_REGION_OPENING, server=asf001.sp2.ygridcore.net,60884,1365336878389, region=da419688e5c79938e207c0d3c056a9ed, current state from region state map ={test1365336888306,,1365336888309.da419688e5c79938e207c0d3c056a9ed. state=OPENING, ts=1365336888410, server=asf001.sp2.ygridcore.net,60884,1365336878389} 2013-04-07 12:14:48,428 INFO [hbase-am-zkevent-worker-pool-20-thread-3] master.RegionStates(264): Region {NAME => 'test1365336888306,,1365336888309.da419688e5c79938e207c0d3c056a9ed.', STARTKEY => '', ENDKEY => '', ENCODED => da419688e5c79938e207c0d3c056a9ed,} transitioned from {test1365336888306,,1365336888309.da419688e5c79938e207c0d3c056a9ed. state=OPENING, ts=1365336888410, server=asf001.sp2.ygridcore.net,60884,1365336878389} to {test1365336888306,,1365336888309.da419688e5c79938e207c0d3c056a9ed. state=OPENING, ts=1365336888428, server=asf001.sp2.ygridcore.net,60884,1365336878389} 2013-04-07 12:14:48,428 DEBUG [PostOpenDeployTasks:da419688e5c79938e207c0d3c056a9ed] zookeeper.ZKUtil(1680): hconnection-0x43ad4a-0x13de46b5f8b0005 Retrieved 43 byte(s) of data from znode /hbase/meta-region-server; data=asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:14:48,428 DEBUG [hbase-am-zkevent-worker-pool-2-thread-19] zookeeper.ZKUtil(1680): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Retrieved 120 byte(s) of data from znode /hbase/region-in-transition/da419688e5c79938e207c0d3c056a9ed and set watcher; region=test1365336888306,,1365336888309.da419688e5c79938e207c0d3c056a9ed., state=RS_ZK_REGION_OPENING, servername=asf001.sp2.ygridcore.net,60884,1365336878389, createTime=1365336888421, payload.length=0 2013-04-07 12:14:48,431 DEBUG [hbase-am-zkevent-worker-pool-2-thread-19] master.AssignmentManager(740): Handling transition=RS_ZK_REGION_OPENING, server=asf001.sp2.ygridcore.net,60884,1365336878389, region=da419688e5c79938e207c0d3c056a9ed, current state from region state map ={test1365336888306,,1365336888309.da419688e5c79938e207c0d3c056a9ed. state=OPENING, ts=1365336888415, server=asf001.sp2.ygridcore.net,60884,1365336878389} 2013-04-07 12:14:48,433 INFO [hbase-am-zkevent-worker-pool-2-thread-19] master.RegionStates(264): Region {NAME => 'test1365336888306,,1365336888309.da419688e5c79938e207c0d3c056a9ed.', STARTKEY => '', ENDKEY => '', ENCODED => da419688e5c79938e207c0d3c056a9ed,} transitioned from {test1365336888306,,1365336888309.da419688e5c79938e207c0d3c056a9ed. state=OPENING, ts=1365336888415, server=asf001.sp2.ygridcore.net,60884,1365336878389} to {test1365336888306,,1365336888309.da419688e5c79938e207c0d3c056a9ed. state=OPENING, ts=1365336888433, server=asf001.sp2.ygridcore.net,60884,1365336878389} 2013-04-07 12:14:48,435 INFO [PostOpenDeployTasks:da419688e5c79938e207c0d3c056a9ed] catalog.MetaEditor(432): Updated row test1365336888306,,1365336888309.da419688e5c79938e207c0d3c056a9ed. with server=asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:14:48,436 INFO [PostOpenDeployTasks:da419688e5c79938e207c0d3c056a9ed] regionserver.HRegionServer(1584): Done with post open deploy task for region=test1365336888306,,1365336888309.da419688e5c79938e207c0d3c056a9ed. 2013-04-07 12:14:48,436 DEBUG [RS_OPEN_REGION-asf001.sp2.ygridcore.net,60884,1365336878389-1] zookeeper.ZKAssign(721): regionserver:60884-0x13de46b5f8b0001 Attempting to transition node da419688e5c79938e207c0d3c056a9ed from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENED 2013-04-07 12:14:48,439 DEBUG [RS_OPEN_REGION-asf001.sp2.ygridcore.net,60884,1365336878389-1] zookeeper.ZKUtil(1680): regionserver:60884-0x13de46b5f8b0001 Retrieved 120 byte(s) of data from znode /hbase/region-in-transition/da419688e5c79938e207c0d3c056a9ed; data=region=test1365336888306,,1365336888309.da419688e5c79938e207c0d3c056a9ed., state=RS_ZK_REGION_OPENING, servername=asf001.sp2.ygridcore.net,60884,1365336878389, createTime=1365336888421, payload.length=0 2013-04-07 12:14:48,441 DEBUG [Thread-200-EventThread] zookeeper.ZooKeeperWatcher(274): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/region-in-transition/da419688e5c79938e207c0d3c056a9ed 2013-04-07 12:14:48,441 DEBUG [RS_OPEN_REGION-asf001.sp2.ygridcore.net,60884,1365336878389-1] zookeeper.ZKAssign(784): regionserver:60884-0x13de46b5f8b0001 Successfully transitioned node da419688e5c79938e207c0d3c056a9ed from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENED 2013-04-07 12:14:48,442 DEBUG [RS_OPEN_REGION-asf001.sp2.ygridcore.net,60884,1365336878389-1] handler.OpenRegionHandler(361): region transitioned to opened in zookeeper: {NAME => 'test1365336888306,,1365336888309.da419688e5c79938e207c0d3c056a9ed.', STARTKEY => '', ENDKEY => '', ENCODED => da419688e5c79938e207c0d3c056a9ed,}, server: asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:14:48,442 DEBUG [RS_OPEN_REGION-asf001.sp2.ygridcore.net,60884,1365336878389-1] handler.OpenRegionHandler(168): Opened test1365336888306,,1365336888309.da419688e5c79938e207c0d3c056a9ed. on server:asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:14:48,443 DEBUG [hbase-am-zkevent-worker-pool-20-thread-4] zookeeper.ZKUtil(1680): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Retrieved 120 byte(s) of data from znode /hbase/region-in-transition/da419688e5c79938e207c0d3c056a9ed and set watcher; region=test1365336888306,,1365336888309.da419688e5c79938e207c0d3c056a9ed., state=RS_ZK_REGION_OPENED, servername=asf001.sp2.ygridcore.net,60884,1365336878389, createTime=1365336888439, payload.length=0 2013-04-07 12:14:48,443 DEBUG [hbase-am-zkevent-worker-pool-20-thread-4] master.AssignmentManager(740): Handling transition=RS_ZK_REGION_OPENED, server=asf001.sp2.ygridcore.net,60884,1365336878389, region=da419688e5c79938e207c0d3c056a9ed, current state from region state map ={test1365336888306,,1365336888309.da419688e5c79938e207c0d3c056a9ed. state=OPENING, ts=1365336888428, server=asf001.sp2.ygridcore.net,60884,1365336878389} 2013-04-07 12:14:48,443 INFO [hbase-am-zkevent-worker-pool-20-thread-4] master.RegionStates(264): Region {NAME => 'test1365336888306,,1365336888309.da419688e5c79938e207c0d3c056a9ed.', STARTKEY => '', ENDKEY => '', ENCODED => da419688e5c79938e207c0d3c056a9ed,} transitioned from {test1365336888306,,1365336888309.da419688e5c79938e207c0d3c056a9ed. state=OPENING, ts=1365336888428, server=asf001.sp2.ygridcore.net,60884,1365336878389} to {test1365336888306,,1365336888309.da419688e5c79938e207c0d3c056a9ed. state=OPEN, ts=1365336888443, server=asf001.sp2.ygridcore.net,60884,1365336878389} 2013-04-07 12:14:48,443 DEBUG [hbase-am-zkevent-worker-pool-2-thread-20] zookeeper.ZKUtil(1680): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Retrieved 120 byte(s) of data from znode /hbase/region-in-transition/da419688e5c79938e207c0d3c056a9ed and set watcher; region=test1365336888306,,1365336888309.da419688e5c79938e207c0d3c056a9ed., state=RS_ZK_REGION_OPENED, servername=asf001.sp2.ygridcore.net,60884,1365336878389, createTime=1365336888439, payload.length=0 2013-04-07 12:14:48,444 DEBUG [MASTER_OPEN_REGION-asf001.sp2.ygridcore.net,33796,1365336878036-3] handler.OpenedRegionHandler(146): Handling OPENED event for test1365336888306,,1365336888309.da419688e5c79938e207c0d3c056a9ed. from asf001.sp2.ygridcore.net,60884,1365336878389; deleting unassigned node 2013-04-07 12:14:48,444 DEBUG [hbase-am-zkevent-worker-pool-2-thread-20] master.AssignmentManager(740): Handling transition=RS_ZK_REGION_OPENED, server=asf001.sp2.ygridcore.net,60884,1365336878389, region=da419688e5c79938e207c0d3c056a9ed, current state from region state map ={test1365336888306,,1365336888309.da419688e5c79938e207c0d3c056a9ed. state=OPENING, ts=1365336888433, server=asf001.sp2.ygridcore.net,60884,1365336878389} 2013-04-07 12:14:48,444 DEBUG [MASTER_OPEN_REGION-asf001.sp2.ygridcore.net,33796,1365336878036-3] zookeeper.ZKAssign(405): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Deleting existing unassigned node for da419688e5c79938e207c0d3c056a9ed that is in expected state RS_ZK_REGION_OPENED 2013-04-07 12:14:48,444 INFO [hbase-am-zkevent-worker-pool-2-thread-20] master.RegionStates(264): Region {NAME => 'test1365336888306,,1365336888309.da419688e5c79938e207c0d3c056a9ed.', STARTKEY => '', ENDKEY => '', ENCODED => da419688e5c79938e207c0d3c056a9ed,} transitioned from {test1365336888306,,1365336888309.da419688e5c79938e207c0d3c056a9ed. state=OPENING, ts=1365336888433, server=asf001.sp2.ygridcore.net,60884,1365336878389} to {test1365336888306,,1365336888309.da419688e5c79938e207c0d3c056a9ed. state=OPEN, ts=1365336888444, server=asf001.sp2.ygridcore.net,60884,1365336878389} 2013-04-07 12:14:48,445 DEBUG [MASTER_OPEN_REGION-asf001.sp2.ygridcore.net,33796,1365336878036-4] handler.OpenedRegionHandler(146): Handling OPENED event for test1365336888306,,1365336888309.da419688e5c79938e207c0d3c056a9ed. from asf001.sp2.ygridcore.net,60884,1365336878389; deleting unassigned node 2013-04-07 12:14:48,445 DEBUG [MASTER_OPEN_REGION-asf001.sp2.ygridcore.net,33796,1365336878036-4] zookeeper.ZKAssign(405): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Deleting existing unassigned node for da419688e5c79938e207c0d3c056a9ed that is in expected state RS_ZK_REGION_OPENED 2013-04-07 12:14:48,446 DEBUG [MASTER_OPEN_REGION-asf001.sp2.ygridcore.net,33796,1365336878036-3] zookeeper.ZKUtil(1680): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Retrieved 120 byte(s) of data from znode /hbase/region-in-transition/da419688e5c79938e207c0d3c056a9ed; data=region=test1365336888306,,1365336888309.da419688e5c79938e207c0d3c056a9ed., state=RS_ZK_REGION_OPENED, servername=asf001.sp2.ygridcore.net,60884,1365336878389, createTime=1365336888439, payload.length=0 2013-04-07 12:14:48,448 DEBUG [Thread-200-EventThread] zookeeper.ZooKeeperWatcher(274): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/region-in-transition/da419688e5c79938e207c0d3c056a9ed 2013-04-07 12:14:48,448 DEBUG [hbase-am-zkevent-worker-pool-20-thread-5] master.AssignmentManager$4(1159): The znode of region test1365336888306,,1365336888309.da419688e5c79938e207c0d3c056a9ed. has been deleted. 2013-04-07 12:14:48,448 DEBUG [hbase-am-zkevent-worker-pool-2-thread-1] master.AssignmentManager$4(1159): The znode of region test1365336888306,,1365336888309.da419688e5c79938e207c0d3c056a9ed. has been deleted. 2013-04-07 12:14:48,448 DEBUG [Thread-200-EventThread] zookeeper.ZooKeeperWatcher(274): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/region-in-transition 2013-04-07 12:14:48,448 INFO [hbase-am-zkevent-worker-pool-2-thread-1] master.RegionStates(264): Region {NAME => 'test1365336888306,,1365336888309.da419688e5c79938e207c0d3c056a9ed.', STARTKEY => '', ENDKEY => '', ENCODED => da419688e5c79938e207c0d3c056a9ed,} transitioned from {test1365336888306,,1365336888309.da419688e5c79938e207c0d3c056a9ed. state=OPEN, ts=1365336888444, server=asf001.sp2.ygridcore.net,60884,1365336878389} to {test1365336888306,,1365336888309.da419688e5c79938e207c0d3c056a9ed. state=OPEN, ts=1365336888448, server=asf001.sp2.ygridcore.net,60884,1365336878389} 2013-04-07 12:14:48,449 DEBUG [MASTER_OPEN_REGION-asf001.sp2.ygridcore.net,33796,1365336878036-3] zookeeper.ZKAssign(434): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Successfully deleted unassigned node for region da419688e5c79938e207c0d3c056a9ed in expected state RS_ZK_REGION_OPENED 2013-04-07 12:14:48,449 INFO [hbase-am-zkevent-worker-pool-2-thread-1] master.RegionStates(299): Onlined region {NAME => 'test1365336888306,,1365336888309.da419688e5c79938e207c0d3c056a9ed.', STARTKEY => '', ENDKEY => '', ENCODED => da419688e5c79938e207c0d3c056a9ed,} on asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:14:48,448 INFO [hbase-am-zkevent-worker-pool-20-thread-5] master.RegionStates(264): Region {NAME => 'test1365336888306,,1365336888309.da419688e5c79938e207c0d3c056a9ed.', STARTKEY => '', ENDKEY => '', ENCODED => da419688e5c79938e207c0d3c056a9ed,} transitioned from {test1365336888306,,1365336888309.da419688e5c79938e207c0d3c056a9ed. state=OPEN, ts=1365336888443, server=asf001.sp2.ygridcore.net,60884,1365336878389} to {test1365336888306,,1365336888309.da419688e5c79938e207c0d3c056a9ed. state=OPEN, ts=1365336888448, server=asf001.sp2.ygridcore.net,60884,1365336878389} 2013-04-07 12:14:48,449 INFO [hbase-am-zkevent-worker-pool-2-thread-1] master.AssignmentManager$4(1164): The master has opened the region test1365336888306,,1365336888309.da419688e5c79938e207c0d3c056a9ed. that was online on asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:14:48,449 INFO [hbase-am-zkevent-worker-pool-20-thread-5] master.RegionStates(299): Onlined region {NAME => 'test1365336888306,,1365336888309.da419688e5c79938e207c0d3c056a9ed.', STARTKEY => '', ENDKEY => '', ENCODED => da419688e5c79938e207c0d3c056a9ed,} on asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:14:48,450 INFO [hbase-am-zkevent-worker-pool-20-thread-5] master.AssignmentManager$4(1164): The master has opened the region test1365336888306,,1365336888309.da419688e5c79938e207c0d3c056a9ed. that was online on asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:14:48,449 DEBUG [MASTER_OPEN_REGION-asf001.sp2.ygridcore.net,33796,1365336878036-4] zookeeper.ZKUtil(775): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Unable to get data of znode /hbase/region-in-transition/da419688e5c79938e207c0d3c056a9ed because node does not exist (not necessarily an error) 2013-04-07 12:14:48,450 WARN [MASTER_OPEN_REGION-asf001.sp2.ygridcore.net,33796,1365336878036-4] handler.OpenedRegionHandler(132): The znode of the region test1365336888306,,1365336888309.da419688e5c79938e207c0d3c056a9ed. would have already been deleted 2013-04-07 12:14:48,450 ERROR [MASTER_OPEN_REGION-asf001.sp2.ygridcore.net,33796,1365336878036-4] handler.OpenedRegionHandler(104): The znode of region test1365336888306,,1365336888309.da419688e5c79938e207c0d3c056a9ed. could not be deleted. 2013-04-07 12:14:49,336 DEBUG [Thread-200] client.MetaScanner(198): Scanning .META. starting at row=test1365336888306,,00000000000000 for max=2147483647 rows using hconnection-0xfe04c6 2013-04-07 12:14:49,337 DEBUG [Thread-200] zookeeper.ZKUtil(1680): hconnection-0xfe04c6-0x13de46b5f8b0007 Retrieved 43 byte(s) of data from znode /hbase/meta-region-server; data=asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:14:49,341 DEBUG [Thread-200] hbase.HRegionInfo(954): Current INFO from scan results = {NAME => 'test1365336888306,,1365336888309.da419688e5c79938e207c0d3c056a9ed.', STARTKEY => '', ENDKEY => '', ENCODED => da419688e5c79938e207c0d3c056a9ed,} 2013-04-07 12:14:49,344 DEBUG [Thread-200] zookeeper.ZKUtil(1680): hconnection-0xfe04c6-0x13de46b5f8b0007 Retrieved 6 byte(s) of data from znode /hbase/table/test1365336888306; data=PBUF\x08\x00 2013-04-07 12:14:49,345 DEBUG [Thread-200] zookeeper.ZKUtil(1680): hconnection-0xfe04c6-0x13de46b5f8b0007 Retrieved 43 byte(s) of data from znode /hbase/meta-region-server; data=asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:14:49,346 DEBUG [Thread-200] zookeeper.ZKUtil(1680): hconnection-0xfe04c6-0x13de46b5f8b0007 Retrieved 43 byte(s) of data from znode /hbase/meta-region-server; data=asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:14:49,347 DEBUG [Thread-200] zookeeper.ZKUtil(1680): hconnection-0xfe04c6-0x13de46b5f8b0007 Retrieved 43 byte(s) of data from znode /hbase/meta-region-server; data=asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:14:49,350 DEBUG [Thread-200] client.MetaScanner(251): Current INFO from scan results = {NAME => 'test1365336888306,,1365336888309.da419688e5c79938e207c0d3c056a9ed.', STARTKEY => '', ENDKEY => '', ENCODED => da419688e5c79938e207c0d3c056a9ed,} 2013-04-07 12:14:49,350 DEBUG [Thread-200] client.MetaScanner(198): Scanning .META. starting at row=test1365336888306,,00000000000000 for max=10 rows using hconnection-0xfe04c6 2013-04-07 12:14:49,351 DEBUG [Thread-200] zookeeper.ZKUtil(1680): hconnection-0xfe04c6-0x13de46b5f8b0007 Retrieved 43 byte(s) of data from znode /hbase/meta-region-server; data=asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:14:49,354 DEBUG [Thread-200] client.MetaScanner(251): Current INFO from scan results = {NAME => 'test1365336888306,,1365336888309.da419688e5c79938e207c0d3c056a9ed.', STARTKEY => '', ENDKEY => '', ENCODED => da419688e5c79938e207c0d3c056a9ed,} 2013-04-07 12:14:49,354 DEBUG [Thread-200] client.HConnectionManager$HConnectionImplementation(1433): Cached location for test1365336888306,,1365336888309.da419688e5c79938e207c0d3c056a9ed. is asf001.sp2.ygridcore.net:60884 2013-04-07 12:14:49,356 INFO [Thread-200] hbase.TestZooKeeper(249): Putting table test1365336888306 2013-04-07 12:14:49,377 INFO [pool-1-thread-1] hbase.ResourceChecker(171): after: TestZooKeeper#testMasterZKSessionRecoveryFailure Thread=212 (was 198) - Thread LEAK? -, OpenFileDescriptor=341 (was 334) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=320 (was 340), ProcessCount=175 (was 175), AvailableMemoryMB=7065 (was 6900) - AvailableMemoryMB LEAK? -, ConnectionCount=4 (was 4) 2013-04-07 12:14:49,727 DEBUG [Thread-200-EventThread] zookeeper.ZooKeeperWatcher(274): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2013-04-07 12:14:49,731 DEBUG [Thread-200-EventThread] zookeeper.ZKUtil(422): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Set watcher on existing znode /hbase/rs/asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:14:49,732 DEBUG [Thread-200-EventThread] zookeeper.ZKUtil(422): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Set watcher on existing znode /hbase/rs/asf001.sp2.ygridcore.net,43802,1365336889418 2013-04-07 12:14:49,735 DEBUG [Thread-200-EventThread] zookeeper.ZKUtil(422): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Set watcher on existing znode /hbase/rs/asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:14:49,737 DEBUG [Thread-200-EventThread] zookeeper.ZKUtil(422): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Set watcher on existing znode /hbase/rs/asf001.sp2.ygridcore.net,43802,1365336889418 2013-04-07 12:14:50,444 DEBUG [Thread-200-EventThread] zookeeper.ZooKeeperWatcher(274): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/region-in-transition 2013-04-07 12:14:50,445 DEBUG [Thread-200-EventThread] master.OfflineCallback(69): rs={testLogSplittingAfterMasterRecoveryDueToZKExpiry,4,1365336889784.567d8ef20f90f3c1b8a889d22a1c3948. state=OFFLINE, ts=1365336890422, server=null}, server=asf001.sp2.ygridcore.net,43802,1365336889418 2013-04-07 12:14:50,453 DEBUG [Thread-200-EventThread] master.OfflineCallback(69): rs={testLogSplittingAfterMasterRecoveryDueToZKExpiry,1,1365336889784.43207168ac66dce85cf2a4f916007450. state=OFFLINE, ts=1365336890422, server=null}, server=asf001.sp2.ygridcore.net,43802,1365336889418 2013-04-07 12:14:50,453 DEBUG [Thread-200-EventThread] master.OfflineCallback(69): rs={testLogSplittingAfterMasterRecoveryDueToZKExpiry,2,1365336889784.55884b67b9203a71bdf8c0e3de70452d. state=OFFLINE, ts=1365336890423, server=null}, server=asf001.sp2.ygridcore.net,43802,1365336889418 2013-04-07 12:14:50,455 DEBUG [Thread-200-EventThread] master.OfflineCallback$ExistCallback(106): rs={testLogSplittingAfterMasterRecoveryDueToZKExpiry,4,1365336889784.567d8ef20f90f3c1b8a889d22a1c3948. state=OFFLINE, ts=1365336890422, server=null}, server=asf001.sp2.ygridcore.net,43802,1365336889418 2013-04-07 12:14:50,456 DEBUG [Thread-200-EventThread] master.OfflineCallback$ExistCallback(106): rs={testLogSplittingAfterMasterRecoveryDueToZKExpiry,1,1365336889784.43207168ac66dce85cf2a4f916007450. state=OFFLINE, ts=1365336890422, server=null}, server=asf001.sp2.ygridcore.net,43802,1365336889418 2013-04-07 12:14:50,456 DEBUG [Thread-200-EventThread] master.OfflineCallback$ExistCallback(106): rs={testLogSplittingAfterMasterRecoveryDueToZKExpiry,2,1365336889784.55884b67b9203a71bdf8c0e3de70452d. state=OFFLINE, ts=1365336890423, server=null}, server=asf001.sp2.ygridcore.net,43802,1365336889418 2013-04-07 12:14:50,457 DEBUG [hbase-am-zkevent-worker-pool-2-thread-3] zookeeper.ZKUtil(1680): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Retrieved 152 byte(s) of data from znode /hbase/region-in-transition/43207168ac66dce85cf2a4f916007450 and set watcher; region=testLogSplittingAfterMasterRecoveryDueToZKExpiry,1,1365336889784.43207168ac66dce85cf2a4f916007450., state=M_ZK_REGION_OFFLINE, servername=asf001.sp2.ygridcore.net,43802,1365336889418, createTime=1365336890424, payload.length=0 2013-04-07 12:14:50,458 DEBUG [hbase-am-zkevent-worker-pool-2-thread-3] zookeeper.ZKUtil(1680): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Retrieved 152 byte(s) of data from znode /hbase/region-in-transition/55884b67b9203a71bdf8c0e3de70452d and set watcher; region=testLogSplittingAfterMasterRecoveryDueToZKExpiry,2,1365336889784.55884b67b9203a71bdf8c0e3de70452d., state=M_ZK_REGION_OFFLINE, servername=asf001.sp2.ygridcore.net,43802,1365336889418, createTime=1365336890424, payload.length=0 2013-04-07 12:14:50,459 DEBUG [hbase-am-zkevent-worker-pool-2-thread-3] zookeeper.ZKUtil(1680): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Retrieved 152 byte(s) of data from znode /hbase/region-in-transition/567d8ef20f90f3c1b8a889d22a1c3948 and set watcher; region=testLogSplittingAfterMasterRecoveryDueToZKExpiry,4,1365336889784.567d8ef20f90f3c1b8a889d22a1c3948., state=M_ZK_REGION_OFFLINE, servername=asf001.sp2.ygridcore.net,43802,1365336889418, createTime=1365336890423, payload.length=0 2013-04-07 12:14:50,460 INFO [MASTER_TABLE_OPERATIONS-asf001.sp2.ygridcore.net,33796,1365336878036-0] master.AssignmentManager(1482): asf001.sp2.ygridcore.net,43802,1365336889418 unassigned znodes=3 of total=3 2013-04-07 12:14:50,460 INFO [MASTER_TABLE_OPERATIONS-asf001.sp2.ygridcore.net,33796,1365336878036-0] master.RegionStates(264): Region {NAME => 'testLogSplittingAfterMasterRecoveryDueToZKExpiry,4,1365336889784.567d8ef20f90f3c1b8a889d22a1c3948.', STARTKEY => '4', ENDKEY => '5', ENCODED => 567d8ef20f90f3c1b8a889d22a1c3948,} transitioned from {testLogSplittingAfterMasterRecoveryDueToZKExpiry,4,1365336889784.567d8ef20f90f3c1b8a889d22a1c3948. state=OFFLINE, ts=1365336890423, server=null} to {testLogSplittingAfterMasterRecoveryDueToZKExpiry,4,1365336889784.567d8ef20f90f3c1b8a889d22a1c3948. state=PENDING_OPEN, ts=1365336890460, server=asf001.sp2.ygridcore.net,43802,1365336889418} 2013-04-07 12:14:50,460 INFO [MASTER_TABLE_OPERATIONS-asf001.sp2.ygridcore.net,33796,1365336878036-0] master.RegionStates(264): Region {NAME => 'testLogSplittingAfterMasterRecoveryDueToZKExpiry,1,1365336889784.43207168ac66dce85cf2a4f916007450.', STARTKEY => '1', ENDKEY => '2', ENCODED => 43207168ac66dce85cf2a4f916007450,} transitioned from {testLogSplittingAfterMasterRecoveryDueToZKExpiry,1,1365336889784.43207168ac66dce85cf2a4f916007450. state=OFFLINE, ts=1365336890423, server=null} to {testLogSplittingAfterMasterRecoveryDueToZKExpiry,1,1365336889784.43207168ac66dce85cf2a4f916007450. state=PENDING_OPEN, ts=1365336890460, server=asf001.sp2.ygridcore.net,43802,1365336889418} 2013-04-07 12:14:50,460 INFO [MASTER_TABLE_OPERATIONS-asf001.sp2.ygridcore.net,33796,1365336878036-0] master.RegionStates(264): Region {NAME => 'testLogSplittingAfterMasterRecoveryDueToZKExpiry,2,1365336889784.55884b67b9203a71bdf8c0e3de70452d.', STARTKEY => '2', ENDKEY => '3', ENCODED => 55884b67b9203a71bdf8c0e3de70452d,} transitioned from {testLogSplittingAfterMasterRecoveryDueToZKExpiry,2,1365336889784.55884b67b9203a71bdf8c0e3de70452d. state=OFFLINE, ts=1365336890424, server=null} to {testLogSplittingAfterMasterRecoveryDueToZKExpiry,2,1365336889784.55884b67b9203a71bdf8c0e3de70452d. state=PENDING_OPEN, ts=1365336890460, server=asf001.sp2.ygridcore.net,43802,1365336889418} 2013-04-07 12:14:50,460 DEBUG [MASTER_TABLE_OPERATIONS-asf001.sp2.ygridcore.net,33796,1365336878036-0] master.ServerManager(713): New connection to asf001.sp2.ygridcore.net,43802,1365336889418 2013-04-07 12:14:50,473 DEBUG [Thread-200-EventThread] zookeeper.ZooKeeperWatcher(274): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/region-in-transition 2013-04-07 12:14:50,474 DEBUG [Thread-200-EventThread] master.OfflineCallback(69): rs={testLogSplittingAfterMasterRecoveryDueToZKExpiry,,1365336889784.fb4182aef4ce07f011871ae0a083aee0. state=OFFLINE, ts=1365336890422, server=null}, server=asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:14:50,476 DEBUG [Thread-200-EventThread] master.OfflineCallback(69): rs={testLogSplittingAfterMasterRecoveryDueToZKExpiry,3,1365336889784.4e9a3d63237d29e6bf55bf2360834cbe. state=OFFLINE, ts=1365336890422, server=null}, server=asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:14:50,562 DEBUG [Thread-200-EventThread] master.OfflineCallback(69): rs={testLogSplittingAfterMasterRecoveryDueToZKExpiry,5,1365336889784.5403a557a6168e5f3c5141eb494f6088. state=OFFLINE, ts=1365336890423, server=null}, server=asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:14:50,564 DEBUG [Thread-200-EventThread] zookeeper.ZooKeeperWatcher(274): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/region-in-transition/567d8ef20f90f3c1b8a889d22a1c3948 2013-04-07 12:14:50,566 DEBUG [Thread-200-EventThread] master.OfflineCallback$ExistCallback(106): rs={testLogSplittingAfterMasterRecoveryDueToZKExpiry,,1365336889784.fb4182aef4ce07f011871ae0a083aee0. state=OFFLINE, ts=1365336890422, server=null}, server=asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:14:50,567 DEBUG [Thread-200-EventThread] zookeeper.ZooKeeperWatcher(274): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/region-in-transition/43207168ac66dce85cf2a4f916007450 2013-04-07 12:14:50,567 DEBUG [Thread-200-EventThread] master.OfflineCallback$ExistCallback(106): rs={testLogSplittingAfterMasterRecoveryDueToZKExpiry,3,1365336889784.4e9a3d63237d29e6bf55bf2360834cbe. state=OFFLINE, ts=1365336890422, server=null}, server=asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:14:50,568 DEBUG [Thread-200-EventThread] zookeeper.ZooKeeperWatcher(274): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/region-in-transition/55884b67b9203a71bdf8c0e3de70452d 2013-04-07 12:14:50,569 DEBUG [Thread-200-EventThread] master.OfflineCallback$ExistCallback(106): rs={testLogSplittingAfterMasterRecoveryDueToZKExpiry,5,1365336889784.5403a557a6168e5f3c5141eb494f6088. state=OFFLINE, ts=1365336890423, server=null}, server=asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:14:50,569 DEBUG [hbase-am-zkevent-worker-pool-20-thread-9] zookeeper.ZKUtil(1680): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Retrieved 152 byte(s) of data from znode /hbase/region-in-transition/567d8ef20f90f3c1b8a889d22a1c3948 and set watcher; region=testLogSplittingAfterMasterRecoveryDueToZKExpiry,4,1365336889784.567d8ef20f90f3c1b8a889d22a1c3948., state=RS_ZK_REGION_OPENING, servername=asf001.sp2.ygridcore.net,43802,1365336889418, createTime=1365336890473, payload.length=0 2013-04-07 12:14:50,569 DEBUG [hbase-am-zkevent-worker-pool-20-thread-9] master.AssignmentManager(740): Handling transition=RS_ZK_REGION_OPENING, server=asf001.sp2.ygridcore.net,43802,1365336889418, region=567d8ef20f90f3c1b8a889d22a1c3948, current state from region state map ={testLogSplittingAfterMasterRecoveryDueToZKExpiry,4,1365336889784.567d8ef20f90f3c1b8a889d22a1c3948. state=PENDING_OPEN, ts=1365336890460, server=asf001.sp2.ygridcore.net,43802,1365336889418} 2013-04-07 12:14:50,570 INFO [hbase-am-zkevent-worker-pool-20-thread-9] master.RegionStates(264): Region {NAME => 'testLogSplittingAfterMasterRecoveryDueToZKExpiry,4,1365336889784.567d8ef20f90f3c1b8a889d22a1c3948.', STARTKEY => '4', ENDKEY => '5', ENCODED => 567d8ef20f90f3c1b8a889d22a1c3948,} transitioned from {testLogSplittingAfterMasterRecoveryDueToZKExpiry,4,1365336889784.567d8ef20f90f3c1b8a889d22a1c3948. state=PENDING_OPEN, ts=1365336890460, server=asf001.sp2.ygridcore.net,43802,1365336889418} to {testLogSplittingAfterMasterRecoveryDueToZKExpiry,4,1365336889784.567d8ef20f90f3c1b8a889d22a1c3948. state=OPENING, ts=1365336890570, server=asf001.sp2.ygridcore.net,43802,1365336889418} 2013-04-07 12:14:50,570 DEBUG [hbase-am-zkevent-worker-pool-2-thread-5] zookeeper.ZKUtil(1680): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Retrieved 152 byte(s) of data from znode /hbase/region-in-transition/567d8ef20f90f3c1b8a889d22a1c3948 and set watcher; region=testLogSplittingAfterMasterRecoveryDueToZKExpiry,4,1365336889784.567d8ef20f90f3c1b8a889d22a1c3948., state=RS_ZK_REGION_OPENING, servername=asf001.sp2.ygridcore.net,43802,1365336889418, createTime=1365336890473, payload.length=0 2013-04-07 12:14:50,571 DEBUG [hbase-am-zkevent-worker-pool-20-thread-10] zookeeper.ZKUtil(1680): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Retrieved 152 byte(s) of data from znode /hbase/region-in-transition/43207168ac66dce85cf2a4f916007450 and set watcher; region=testLogSplittingAfterMasterRecoveryDueToZKExpiry,1,1365336889784.43207168ac66dce85cf2a4f916007450., state=RS_ZK_REGION_OPENING, servername=asf001.sp2.ygridcore.net,43802,1365336889418, createTime=1365336890475, payload.length=0 2013-04-07 12:14:50,572 DEBUG [hbase-am-zkevent-worker-pool-20-thread-10] master.AssignmentManager(740): Handling transition=RS_ZK_REGION_OPENING, server=asf001.sp2.ygridcore.net,43802,1365336889418, region=43207168ac66dce85cf2a4f916007450, current state from region state map ={testLogSplittingAfterMasterRecoveryDueToZKExpiry,1,1365336889784.43207168ac66dce85cf2a4f916007450. state=PENDING_OPEN, ts=1365336890460, server=asf001.sp2.ygridcore.net,43802,1365336889418} 2013-04-07 12:14:50,572 INFO [hbase-am-zkevent-worker-pool-20-thread-10] master.RegionStates(264): Region {NAME => 'testLogSplittingAfterMasterRecoveryDueToZKExpiry,1,1365336889784.43207168ac66dce85cf2a4f916007450.', STARTKEY => '1', ENDKEY => '2', ENCODED => 43207168ac66dce85cf2a4f916007450,} transitioned from {testLogSplittingAfterMasterRecoveryDueToZKExpiry,1,1365336889784.43207168ac66dce85cf2a4f916007450. state=PENDING_OPEN, ts=1365336890460, server=asf001.sp2.ygridcore.net,43802,1365336889418} to {testLogSplittingAfterMasterRecoveryDueToZKExpiry,1,1365336889784.43207168ac66dce85cf2a4f916007450. state=OPENING, ts=1365336890572, server=asf001.sp2.ygridcore.net,43802,1365336889418} 2013-04-07 12:14:50,572 DEBUG [hbase-am-zkevent-worker-pool-2-thread-6] zookeeper.ZKUtil(1680): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Retrieved 152 byte(s) of data from znode /hbase/region-in-transition/43207168ac66dce85cf2a4f916007450 and set watcher; region=testLogSplittingAfterMasterRecoveryDueToZKExpiry,1,1365336889784.43207168ac66dce85cf2a4f916007450., state=RS_ZK_REGION_OPENING, servername=asf001.sp2.ygridcore.net,43802,1365336889418, createTime=1365336890475, payload.length=0 2013-04-07 12:14:50,572 DEBUG [hbase-am-zkevent-worker-pool-2-thread-5] zookeeper.ZKUtil(1680): hconnection-0x1f327e-0x13de46b5f8b0003 Retrieved 43 byte(s) of data from znode /hbase/meta-region-server; data=asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:14:50,573 DEBUG [hbase-am-zkevent-worker-pool-20-thread-11] zookeeper.ZKUtil(1680): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Retrieved 152 byte(s) of data from znode /hbase/region-in-transition/55884b67b9203a71bdf8c0e3de70452d and set watcher; region=testLogSplittingAfterMasterRecoveryDueToZKExpiry,2,1365336889784.55884b67b9203a71bdf8c0e3de70452d., state=RS_ZK_REGION_OPENING, servername=asf001.sp2.ygridcore.net,43802,1365336889418, createTime=1365336890477, payload.length=0 2013-04-07 12:14:50,573 DEBUG [hbase-am-zkevent-worker-pool-20-thread-11] master.AssignmentManager(740): Handling transition=RS_ZK_REGION_OPENING, server=asf001.sp2.ygridcore.net,43802,1365336889418, region=55884b67b9203a71bdf8c0e3de70452d, current state from region state map ={testLogSplittingAfterMasterRecoveryDueToZKExpiry,2,1365336889784.55884b67b9203a71bdf8c0e3de70452d. state=PENDING_OPEN, ts=1365336890460, server=asf001.sp2.ygridcore.net,43802,1365336889418} 2013-04-07 12:14:50,573 INFO [MASTER_TABLE_OPERATIONS-asf001.sp2.ygridcore.net,33796,1365336878036-0] master.AssignmentManager(1482): asf001.sp2.ygridcore.net,60884,1365336878389 unassigned znodes=3 of total=3 2013-04-07 12:14:50,573 INFO [hbase-am-zkevent-worker-pool-20-thread-11] master.RegionStates(264): Region {NAME => 'testLogSplittingAfterMasterRecoveryDueToZKExpiry,2,1365336889784.55884b67b9203a71bdf8c0e3de70452d.', STARTKEY => '2', ENDKEY => '3', ENCODED => 55884b67b9203a71bdf8c0e3de70452d,} transitioned from {testLogSplittingAfterMasterRecoveryDueToZKExpiry,2,1365336889784.55884b67b9203a71bdf8c0e3de70452d. state=PENDING_OPEN, ts=1365336890460, server=asf001.sp2.ygridcore.net,43802,1365336889418} to {testLogSplittingAfterMasterRecoveryDueToZKExpiry,2,1365336889784.55884b67b9203a71bdf8c0e3de70452d. state=OPENING, ts=1365336890573, server=asf001.sp2.ygridcore.net,43802,1365336889418} 2013-04-07 12:14:50,574 INFO [MASTER_TABLE_OPERATIONS-asf001.sp2.ygridcore.net,33796,1365336878036-0] master.RegionStates(264): Region {NAME => 'testLogSplittingAfterMasterRecoveryDueToZKExpiry,,1365336889784.fb4182aef4ce07f011871ae0a083aee0.', STARTKEY => '', ENDKEY => '1', ENCODED => fb4182aef4ce07f011871ae0a083aee0,} transitioned from {testLogSplittingAfterMasterRecoveryDueToZKExpiry,,1365336889784.fb4182aef4ce07f011871ae0a083aee0. state=OFFLINE, ts=1365336890470, server=null} to {testLogSplittingAfterMasterRecoveryDueToZKExpiry,,1365336889784.fb4182aef4ce07f011871ae0a083aee0. state=PENDING_OPEN, ts=1365336890574, server=asf001.sp2.ygridcore.net,60884,1365336878389} 2013-04-07 12:14:50,574 DEBUG [hbase-am-zkevent-worker-pool-2-thread-7] zookeeper.ZKUtil(1680): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Retrieved 152 byte(s) of data from znode /hbase/region-in-transition/55884b67b9203a71bdf8c0e3de70452d and set watcher; region=testLogSplittingAfterMasterRecoveryDueToZKExpiry,2,1365336889784.55884b67b9203a71bdf8c0e3de70452d., state=RS_ZK_REGION_OPENING, servername=asf001.sp2.ygridcore.net,43802,1365336889418, createTime=1365336890477, payload.length=0 2013-04-07 12:14:50,574 INFO [MASTER_TABLE_OPERATIONS-asf001.sp2.ygridcore.net,33796,1365336878036-0] master.RegionStates(264): Region {NAME => 'testLogSplittingAfterMasterRecoveryDueToZKExpiry,3,1365336889784.4e9a3d63237d29e6bf55bf2360834cbe.', STARTKEY => '3', ENDKEY => '4', ENCODED => 4e9a3d63237d29e6bf55bf2360834cbe,} transitioned from {testLogSplittingAfterMasterRecoveryDueToZKExpiry,3,1365336889784.4e9a3d63237d29e6bf55bf2360834cbe. state=OFFLINE, ts=1365336890471, server=null} to {testLogSplittingAfterMasterRecoveryDueToZKExpiry,3,1365336889784.4e9a3d63237d29e6bf55bf2360834cbe. state=PENDING_OPEN, ts=1365336890574, server=asf001.sp2.ygridcore.net,60884,1365336878389} 2013-04-07 12:14:50,575 INFO [MASTER_TABLE_OPERATIONS-asf001.sp2.ygridcore.net,33796,1365336878036-0] master.RegionStates(264): Region {NAME => 'testLogSplittingAfterMasterRecoveryDueToZKExpiry,5,1365336889784.5403a557a6168e5f3c5141eb494f6088.', STARTKEY => '5', ENDKEY => '', ENCODED => 5403a557a6168e5f3c5141eb494f6088,} transitioned from {testLogSplittingAfterMasterRecoveryDueToZKExpiry,5,1365336889784.5403a557a6168e5f3c5141eb494f6088. state=OFFLINE, ts=1365336890471, server=null} to {testLogSplittingAfterMasterRecoveryDueToZKExpiry,5,1365336889784.5403a557a6168e5f3c5141eb494f6088. state=PENDING_OPEN, ts=1365336890575, server=asf001.sp2.ygridcore.net,60884,1365336878389} 2013-04-07 12:14:50,576 DEBUG [hbase-am-zkevent-worker-pool-2-thread-5] zookeeper.ZKUtil(1680): hconnection-0x1f327e-0x13de46b5f8b0003 Retrieved 43 byte(s) of data from znode /hbase/meta-region-server; data=asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:14:50,576 INFO [PRI IPC Server handler 1 on 60884] regionserver.HRegionServer(3346): Received request to open region: testLogSplittingAfterMasterRecoveryDueToZKExpiry,,1365336889784.fb4182aef4ce07f011871ae0a083aee0. on asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:14:50,579 INFO [hbase-am-zkevent-worker-pool-2-thread-5] master.RegionStates(264): Region {NAME => 'testLogSplittingAfterMasterRecoveryDueToZKExpiry,4,1365336889784.567d8ef20f90f3c1b8a889d22a1c3948.', STARTKEY => '4', ENDKEY => '5', ENCODED => 567d8ef20f90f3c1b8a889d22a1c3948,} transitioned from {testLogSplittingAfterMasterRecoveryDueToZKExpiry,4,1365336889784.567d8ef20f90f3c1b8a889d22a1c3948. state=OFFLINE, ts=1365336890579, server=null} to {testLogSplittingAfterMasterRecoveryDueToZKExpiry,4,1365336889784.567d8ef20f90f3c1b8a889d22a1c3948. state=OPENING, ts=1365336890579, server=asf001.sp2.ygridcore.net,43802,1365336889418} 2013-04-07 12:14:50,579 DEBUG [hbase-am-zkevent-worker-pool-2-thread-7] master.AssignmentManager(740): Handling transition=RS_ZK_REGION_OPENING, server=asf001.sp2.ygridcore.net,43802,1365336889418, region=55884b67b9203a71bdf8c0e3de70452d, current state from region state map =null 2013-04-07 12:14:50,579 DEBUG [hbase-am-zkevent-worker-pool-2-thread-6] master.AssignmentManager(740): Handling transition=RS_ZK_REGION_OPENING, server=asf001.sp2.ygridcore.net,43802,1365336889418, region=43207168ac66dce85cf2a4f916007450, current state from region state map =null 2013-04-07 12:14:50,582 DEBUG [hbase-am-zkevent-worker-pool-2-thread-4] zookeeper.ZKUtil(1680): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Retrieved 151 byte(s) of data from znode /hbase/region-in-transition/fb4182aef4ce07f011871ae0a083aee0 and set watcher; region=testLogSplittingAfterMasterRecoveryDueToZKExpiry,,1365336889784.fb4182aef4ce07f011871ae0a083aee0., state=M_ZK_REGION_OFFLINE, servername=asf001.sp2.ygridcore.net,60884,1365336878389, createTime=1365336890471, payload.length=0 2013-04-07 12:14:50,582 INFO [PRI IPC Server handler 1 on 60884] regionserver.HRegionServer(3346): Received request to open region: testLogSplittingAfterMasterRecoveryDueToZKExpiry,3,1365336889784.4e9a3d63237d29e6bf55bf2360834cbe. on asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:14:50,583 DEBUG [RS_OPEN_REGION-asf001.sp2.ygridcore.net,60884,1365336878389-0] zookeeper.ZKAssign(721): regionserver:60884-0x13de46b5f8b0001 Attempting to transition node 4e9a3d63237d29e6bf55bf2360834cbe from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING 2013-04-07 12:14:50,583 INFO [PRI IPC Server handler 1 on 60884] regionserver.HRegionServer(3346): Received request to open region: testLogSplittingAfterMasterRecoveryDueToZKExpiry,5,1365336889784.5403a557a6168e5f3c5141eb494f6088. on asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:14:50,583 DEBUG [hbase-am-zkevent-worker-pool-2-thread-7] zookeeper.ZKUtil(1680): hconnection-0x1f327e-0x13de46b5f8b0003 Retrieved 43 byte(s) of data from znode /hbase/meta-region-server; data=asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:14:50,600 DEBUG [Thread-200-EventThread] zookeeper.ZooKeeperWatcher(274): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/region-in-transition/4e9a3d63237d29e6bf55bf2360834cbe 2013-04-07 12:14:50,601 DEBUG [RS_OPEN_REGION-asf001.sp2.ygridcore.net,60884,1365336878389-0] zookeeper.ZKAssign(784): regionserver:60884-0x13de46b5f8b0001 Successfully transitioned node 4e9a3d63237d29e6bf55bf2360834cbe from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING 2013-04-07 12:14:50,601 INFO [RS_OPEN_REGION-asf001.sp2.ygridcore.net,60884,1365336878389-0] regionserver.HRegion(4023): HRegion.openHRegion Region name ==testLogSplittingAfterMasterRecoveryDueToZKExpiry,3,1365336889784.4e9a3d63237d29e6bf55bf2360834cbe. 2013-04-07 12:14:50,601 DEBUG [RS_OPEN_REGION-asf001.sp2.ygridcore.net,60884,1365336878389-0] regionserver.HRegion(4025): Opening region: {NAME => 'testLogSplittingAfterMasterRecoveryDueToZKExpiry,3,1365336889784.4e9a3d63237d29e6bf55bf2360834cbe.', STARTKEY => '3', ENDKEY => '4', ENCODED => 4e9a3d63237d29e6bf55bf2360834cbe,} 2013-04-07 12:14:50,608 DEBUG [Thread-200-EventThread] zookeeper.ZooKeeperWatcher(274): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/region-in-transition/fb4182aef4ce07f011871ae0a083aee0 2013-04-07 12:14:50,610 DEBUG [Thread-200-EventThread] zookeeper.ZooKeeperWatcher(274): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/region-in-transition/5403a557a6168e5f3c5141eb494f6088 2013-04-07 12:14:50,612 DEBUG [hbase-am-zkevent-worker-pool-20-thread-12] zookeeper.ZKUtil(1680): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Retrieved 152 byte(s) of data from znode /hbase/region-in-transition/4e9a3d63237d29e6bf55bf2360834cbe and set watcher; region=testLogSplittingAfterMasterRecoveryDueToZKExpiry,3,1365336889784.4e9a3d63237d29e6bf55bf2360834cbe., state=RS_ZK_REGION_OPENING, servername=asf001.sp2.ygridcore.net,60884,1365336878389, createTime=1365336890587, payload.length=0 2013-04-07 12:14:50,613 DEBUG [hbase-am-zkevent-worker-pool-20-thread-12] master.AssignmentManager(740): Handling transition=RS_ZK_REGION_OPENING, server=asf001.sp2.ygridcore.net,60884,1365336878389, region=4e9a3d63237d29e6bf55bf2360834cbe, current state from region state map ={testLogSplittingAfterMasterRecoveryDueToZKExpiry,3,1365336889784.4e9a3d63237d29e6bf55bf2360834cbe. state=PENDING_OPEN, ts=1365336890574, server=asf001.sp2.ygridcore.net,60884,1365336878389} 2013-04-07 12:14:50,613 DEBUG [hbase-am-zkevent-worker-pool-2-thread-8] zookeeper.ZKUtil(1680): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Retrieved 152 byte(s) of data from znode /hbase/region-in-transition/4e9a3d63237d29e6bf55bf2360834cbe and set watcher; region=testLogSplittingAfterMasterRecoveryDueToZKExpiry,3,1365336889784.4e9a3d63237d29e6bf55bf2360834cbe., state=RS_ZK_REGION_OPENING, servername=asf001.sp2.ygridcore.net,60884,1365336878389, createTime=1365336890587, payload.length=0 2013-04-07 12:14:50,613 INFO [hbase-am-zkevent-worker-pool-20-thread-12] master.RegionStates(264): Region {NAME => 'testLogSplittingAfterMasterRecoveryDueToZKExpiry,3,1365336889784.4e9a3d63237d29e6bf55bf2360834cbe.', STARTKEY => '3', ENDKEY => '4', ENCODED => 4e9a3d63237d29e6bf55bf2360834cbe,} transitioned from {testLogSplittingAfterMasterRecoveryDueToZKExpiry,3,1365336889784.4e9a3d63237d29e6bf55bf2360834cbe. state=PENDING_OPEN, ts=1365336890574, server=asf001.sp2.ygridcore.net,60884,1365336878389} to {testLogSplittingAfterMasterRecoveryDueToZKExpiry,3,1365336889784.4e9a3d63237d29e6bf55bf2360834cbe. state=OPENING, ts=1365336890613, server=asf001.sp2.ygridcore.net,60884,1365336878389} 2013-04-07 12:14:50,613 DEBUG [hbase-am-zkevent-worker-pool-2-thread-8] master.AssignmentManager(740): Handling transition=RS_ZK_REGION_OPENING, server=asf001.sp2.ygridcore.net,60884,1365336878389, region=4e9a3d63237d29e6bf55bf2360834cbe, current state from region state map =null 2013-04-07 12:14:50,616 DEBUG [Thread-200-EventThread] zookeeper.ZooKeeperWatcher(274): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/region-in-transition/567d8ef20f90f3c1b8a889d22a1c3948 2013-04-07 12:14:50,618 DEBUG [hbase-am-zkevent-worker-pool-2-thread-9] zookeeper.ZKUtil(1680): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Retrieved 151 byte(s) of data from znode /hbase/region-in-transition/fb4182aef4ce07f011871ae0a083aee0 and set watcher; region=testLogSplittingAfterMasterRecoveryDueToZKExpiry,,1365336889784.fb4182aef4ce07f011871ae0a083aee0., state=RS_ZK_REGION_OPENING, servername=asf001.sp2.ygridcore.net,60884,1365336878389, createTime=1365336890590, payload.length=0 2013-04-07 12:14:50,619 DEBUG [hbase-am-zkevent-worker-pool-20-thread-13] zookeeper.ZKUtil(1680): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Retrieved 151 byte(s) of data from znode /hbase/region-in-transition/fb4182aef4ce07f011871ae0a083aee0 and set watcher; region=testLogSplittingAfterMasterRecoveryDueToZKExpiry,,1365336889784.fb4182aef4ce07f011871ae0a083aee0., state=RS_ZK_REGION_OPENING, servername=asf001.sp2.ygridcore.net,60884,1365336878389, createTime=1365336890590, payload.length=0 2013-04-07 12:14:50,619 DEBUG [hbase-am-zkevent-worker-pool-20-thread-13] master.AssignmentManager(740): Handling transition=RS_ZK_REGION_OPENING, server=asf001.sp2.ygridcore.net,60884,1365336878389, region=fb4182aef4ce07f011871ae0a083aee0, current state from region state map ={testLogSplittingAfterMasterRecoveryDueToZKExpiry,,1365336889784.fb4182aef4ce07f011871ae0a083aee0. state=PENDING_OPEN, ts=1365336890574, server=asf001.sp2.ygridcore.net,60884,1365336878389} 2013-04-07 12:14:50,620 INFO [hbase-am-zkevent-worker-pool-20-thread-13] master.RegionStates(264): Region {NAME => 'testLogSplittingAfterMasterRecoveryDueToZKExpiry,,1365336889784.fb4182aef4ce07f011871ae0a083aee0.', STARTKEY => '', ENDKEY => '1', ENCODED => fb4182aef4ce07f011871ae0a083aee0,} transitioned from {testLogSplittingAfterMasterRecoveryDueToZKExpiry,,1365336889784.fb4182aef4ce07f011871ae0a083aee0. state=PENDING_OPEN, ts=1365336890574, server=asf001.sp2.ygridcore.net,60884,1365336878389} to {testLogSplittingAfterMasterRecoveryDueToZKExpiry,,1365336889784.fb4182aef4ce07f011871ae0a083aee0. state=OPENING, ts=1365336890620, server=asf001.sp2.ygridcore.net,60884,1365336878389} 2013-04-07 12:14:50,622 DEBUG [hbase-am-zkevent-worker-pool-20-thread-14] zookeeper.ZKUtil(1680): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Retrieved 152 byte(s) of data from znode /hbase/region-in-transition/5403a557a6168e5f3c5141eb494f6088 and set watcher; region=testLogSplittingAfterMasterRecoveryDueToZKExpiry,5,1365336889784.5403a557a6168e5f3c5141eb494f6088., state=RS_ZK_REGION_OPENING, servername=asf001.sp2.ygridcore.net,60884,1365336878389, createTime=1365336890592, payload.length=0 2013-04-07 12:14:50,623 DEBUG [hbase-am-zkevent-worker-pool-20-thread-14] master.AssignmentManager(740): Handling transition=RS_ZK_REGION_OPENING, server=asf001.sp2.ygridcore.net,60884,1365336878389, region=5403a557a6168e5f3c5141eb494f6088, current state from region state map ={testLogSplittingAfterMasterRecoveryDueToZKExpiry,5,1365336889784.5403a557a6168e5f3c5141eb494f6088. state=PENDING_OPEN, ts=1365336890575, server=asf001.sp2.ygridcore.net,60884,1365336878389} 2013-04-07 12:14:50,623 INFO [StoreOpenerThread-testLogSplittingAfterMasterRecoveryDueToZKExpiry,5,1365336889784.5403a557a6168e5f3c5141eb494f6088.-1] compactions.CompactionConfiguration(85): Compaction configuration size [134217728, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; delete expired; major period 86400000, major jitter 0.200000 2013-04-07 12:14:50,623 INFO [RS_OPEN_REGION-asf001.sp2.ygridcore.net,60884,1365336878389-0] regionserver.HRegion(586): Onlined testLogSplittingAfterMasterRecoveryDueToZKExpiry,3,1365336889784.4e9a3d63237d29e6bf55bf2360834cbe.; next sequenceid=1 2013-04-07 12:14:50,623 DEBUG [RS_OPEN_REGION-asf001.sp2.ygridcore.net,60884,1365336878389-0] zookeeper.ZKAssign(721): regionserver:60884-0x13de46b5f8b0001 Attempting to transition node 4e9a3d63237d29e6bf55bf2360834cbe from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENING 2013-04-07 12:14:50,623 DEBUG [hbase-am-zkevent-worker-pool-2-thread-11] zookeeper.ZKUtil(1680): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Retrieved 152 byte(s) of data from znode /hbase/region-in-transition/567d8ef20f90f3c1b8a889d22a1c3948 and set watcher; region=testLogSplittingAfterMasterRecoveryDueToZKExpiry,4,1365336889784.567d8ef20f90f3c1b8a889d22a1c3948., state=RS_ZK_REGION_OPENING, servername=asf001.sp2.ygridcore.net,43802,1365336889418, createTime=1365336890609, payload.length=0 2013-04-07 12:14:50,624 INFO [hbase-am-zkevent-worker-pool-20-thread-14] master.RegionStates(264): Region {NAME => 'testLogSplittingAfterMasterRecoveryDueToZKExpiry,5,1365336889784.5403a557a6168e5f3c5141eb494f6088.', STARTKEY => '5', ENDKEY => '', ENCODED => 5403a557a6168e5f3c5141eb494f6088,} transitioned from {testLogSplittingAfterMasterRecoveryDueToZKExpiry,5,1365336889784.5403a557a6168e5f3c5141eb494f6088. state=PENDING_OPEN, ts=1365336890575, server=asf001.sp2.ygridcore.net,60884,1365336878389} to {testLogSplittingAfterMasterRecoveryDueToZKExpiry,5,1365336889784.5403a557a6168e5f3c5141eb494f6088. state=OPENING, ts=1365336890623, server=asf001.sp2.ygridcore.net,60884,1365336878389} 2013-04-07 12:14:50,623 DEBUG [MASTER_TABLE_OPERATIONS-asf001.sp2.ygridcore.net,33796,1365336878036-0] lock.ZKInterProcessLockBase(297): Successfully released /hbase/table-lock/testLogSplittingAfterMasterRecoveryDueToZKExpiry/write-master:337960000000000 2013-04-07 12:14:50,624 DEBUG [MASTER_TABLE_OPERATIONS-asf001.sp2.ygridcore.net,33796,1365336878036-0] master.TableLockManager$ZKTableLockManager$TableLockImpl(269): Released table lock on :testLogSplittingAfterMasterRecoveryDueToZKExpiry 2013-04-07 12:14:50,624 DEBUG [hbase-am-zkevent-worker-pool-20-thread-15] zookeeper.ZKUtil(1680): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Retrieved 152 byte(s) of data from znode /hbase/region-in-transition/567d8ef20f90f3c1b8a889d22a1c3948 and set watcher; region=testLogSplittingAfterMasterRecoveryDueToZKExpiry,4,1365336889784.567d8ef20f90f3c1b8a889d22a1c3948., state=RS_ZK_REGION_OPENING, servername=asf001.sp2.ygridcore.net,43802,1365336889418, createTime=1365336890609, payload.length=0 2013-04-07 12:14:50,624 DEBUG [hbase-am-zkevent-worker-pool-20-thread-15] master.AssignmentManager(740): Handling transition=RS_ZK_REGION_OPENING, server=asf001.sp2.ygridcore.net,43802,1365336889418, region=567d8ef20f90f3c1b8a889d22a1c3948, current state from region state map ={testLogSplittingAfterMasterRecoveryDueToZKExpiry,4,1365336889784.567d8ef20f90f3c1b8a889d22a1c3948. state=OPENING, ts=1365336890570, server=asf001.sp2.ygridcore.net,43802,1365336889418} 2013-04-07 12:14:50,624 INFO [hbase-am-zkevent-worker-pool-20-thread-15] master.RegionStates(264): Region {NAME => 'testLogSplittingAfterMasterRecoveryDueToZKExpiry,4,1365336889784.567d8ef20f90f3c1b8a889d22a1c3948.', STARTKEY => '4', ENDKEY => '5', ENCODED => 567d8ef20f90f3c1b8a889d22a1c3948,} transitioned from {testLogSplittingAfterMasterRecoveryDueToZKExpiry,4,1365336889784.567d8ef20f90f3c1b8a889d22a1c3948. state=OPENING, ts=1365336890570, server=asf001.sp2.ygridcore.net,43802,1365336889418} to {testLogSplittingAfterMasterRecoveryDueToZKExpiry,4,1365336889784.567d8ef20f90f3c1b8a889d22a1c3948. state=OPENING, ts=1365336890624, server=asf001.sp2.ygridcore.net,43802,1365336889418} 2013-04-07 12:14:50,625 INFO [hbase-am-zkevent-worker-pool-2-thread-8] master.RegionStates(264): Region {NAME => 'testLogSplittingAfterMasterRecoveryDueToZKExpiry,3,1365336889784.4e9a3d63237d29e6bf55bf2360834cbe.', STARTKEY => '3', ENDKEY => '4', ENCODED => 4e9a3d63237d29e6bf55bf2360834cbe,} transitioned from {testLogSplittingAfterMasterRecoveryDueToZKExpiry,3,1365336889784.4e9a3d63237d29e6bf55bf2360834cbe. state=OFFLINE, ts=1365336890624, server=null} to {testLogSplittingAfterMasterRecoveryDueToZKExpiry,3,1365336889784.4e9a3d63237d29e6bf55bf2360834cbe. state=OPENING, ts=1365336890625, server=asf001.sp2.ygridcore.net,60884,1365336878389} 2013-04-07 12:14:50,625 DEBUG [hbase-am-zkevent-worker-pool-2-thread-11] master.AssignmentManager(740): Handling transition=RS_ZK_REGION_OPENING, server=asf001.sp2.ygridcore.net,43802,1365336889418, region=567d8ef20f90f3c1b8a889d22a1c3948, current state from region state map ={testLogSplittingAfterMasterRecoveryDueToZKExpiry,4,1365336889784.567d8ef20f90f3c1b8a889d22a1c3948. state=OPENING, ts=1365336890579, server=asf001.sp2.ygridcore.net,43802,1365336889418} 2013-04-07 12:14:50,625 INFO [StoreOpenerThread-testLogSplittingAfterMasterRecoveryDueToZKExpiry,,1365336889784.fb4182aef4ce07f011871ae0a083aee0.-1] compactions.CompactionConfiguration(85): Compaction configuration size [134217728, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; delete expired; major period 86400000, major jitter 0.200000 2013-04-07 12:14:50,628 INFO [hbase-am-zkevent-worker-pool-2-thread-11] master.RegionStates(264): Region {NAME => 'testLogSplittingAfterMasterRecoveryDueToZKExpiry,4,1365336889784.567d8ef20f90f3c1b8a889d22a1c3948.', STARTKEY => '4', ENDKEY => '5', ENCODED => 567d8ef20f90f3c1b8a889d22a1c3948,} transitioned from {testLogSplittingAfterMasterRecoveryDueToZKExpiry,4,1365336889784.567d8ef20f90f3c1b8a889d22a1c3948. state=OPENING, ts=1365336890579, server=asf001.sp2.ygridcore.net,43802,1365336889418} to {testLogSplittingAfterMasterRecoveryDueToZKExpiry,4,1365336889784.567d8ef20f90f3c1b8a889d22a1c3948. state=OPENING, ts=1365336890628, server=asf001.sp2.ygridcore.net,43802,1365336889418} 2013-04-07 12:14:50,628 DEBUG [Thread-200-EventThread] zookeeper.ZooKeeperWatcher(274): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/region-in-transition/43207168ac66dce85cf2a4f916007450 2013-04-07 12:14:50,637 DEBUG [hbase-am-zkevent-worker-pool-20-thread-16] zookeeper.ZKUtil(1680): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Retrieved 152 byte(s) of data from znode /hbase/region-in-transition/43207168ac66dce85cf2a4f916007450 and set watcher; region=testLogSplittingAfterMasterRecoveryDueToZKExpiry,1,1365336889784.43207168ac66dce85cf2a4f916007450., state=RS_ZK_REGION_OPENING, servername=asf001.sp2.ygridcore.net,43802,1365336889418, createTime=1365336890618, payload.length=0 2013-04-07 12:14:50,637 INFO [RS_OPEN_REGION-asf001.sp2.ygridcore.net,60884,1365336878389-1] regionserver.HRegion(586): Onlined testLogSplittingAfterMasterRecoveryDueToZKExpiry,5,1365336889784.5403a557a6168e5f3c5141eb494f6088.; next sequenceid=1 2013-04-07 12:14:50,638 DEBUG [hbase-am-zkevent-worker-pool-20-thread-16] master.AssignmentManager(740): Handling transition=RS_ZK_REGION_OPENING, server=asf001.sp2.ygridcore.net,43802,1365336889418, region=43207168ac66dce85cf2a4f916007450, current state from region state map ={testLogSplittingAfterMasterRecoveryDueToZKExpiry,1,1365336889784.43207168ac66dce85cf2a4f916007450. state=OPENING, ts=1365336890572, server=asf001.sp2.ygridcore.net,43802,1365336889418} 2013-04-07 12:14:50,638 DEBUG [RS_OPEN_REGION-asf001.sp2.ygridcore.net,60884,1365336878389-1] zookeeper.ZKAssign(721): regionserver:60884-0x13de46b5f8b0001 Attempting to transition node 5403a557a6168e5f3c5141eb494f6088 from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENING 2013-04-07 12:14:50,638 INFO [hbase-am-zkevent-worker-pool-20-thread-16] master.RegionStates(264): Region {NAME => 'testLogSplittingAfterMasterRecoveryDueToZKExpiry,1,1365336889784.43207168ac66dce85cf2a4f916007450.', STARTKEY => '1', ENDKEY => '2', ENCODED => 43207168ac66dce85cf2a4f916007450,} transitioned from {testLogSplittingAfterMasterRecoveryDueToZKExpiry,1,1365336889784.43207168ac66dce85cf2a4f916007450. state=OPENING, ts=1365336890572, server=asf001.sp2.ygridcore.net,43802,1365336889418} to {testLogSplittingAfterMasterRecoveryDueToZKExpiry,1,1365336889784.43207168ac66dce85cf2a4f916007450. state=OPENING, ts=1365336890638, server=asf001.sp2.ygridcore.net,43802,1365336889418} 2013-04-07 12:14:50,638 DEBUG [hbase-am-zkevent-worker-pool-2-thread-10] zookeeper.ZKUtil(1680): hconnection-0x1f327e-0x13de46b5f8b0003 Retrieved 43 byte(s) of data from znode /hbase/meta-region-server; data=asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:14:50,641 DEBUG [Thread-200-EventThread] zookeeper.ZooKeeperWatcher(274): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/region-in-transition/4e9a3d63237d29e6bf55bf2360834cbe 2013-04-07 12:14:50,650 DEBUG [hbase-am-zkevent-worker-pool-20-thread-17] zookeeper.ZKUtil(1680): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Retrieved 152 byte(s) of data from znode /hbase/region-in-transition/4e9a3d63237d29e6bf55bf2360834cbe and set watcher; region=testLogSplittingAfterMasterRecoveryDueToZKExpiry,3,1365336889784.4e9a3d63237d29e6bf55bf2360834cbe., state=RS_ZK_REGION_OPENING, servername=asf001.sp2.ygridcore.net,60884,1365336878389, createTime=1365336890637, payload.length=0 2013-04-07 12:14:50,650 DEBUG [hbase-am-zkevent-worker-pool-20-thread-17] master.AssignmentManager(740): Handling transition=RS_ZK_REGION_OPENING, server=asf001.sp2.ygridcore.net,60884,1365336878389, region=4e9a3d63237d29e6bf55bf2360834cbe, current state from region state map ={testLogSplittingAfterMasterRecoveryDueToZKExpiry,3,1365336889784.4e9a3d63237d29e6bf55bf2360834cbe. state=OPENING, ts=1365336890613, server=asf001.sp2.ygridcore.net,60884,1365336878389} 2013-04-07 12:14:50,650 DEBUG [hbase-am-zkevent-worker-pool-2-thread-9] zookeeper.ZKUtil(1680): hconnection-0x1f327e-0x13de46b5f8b0003 Retrieved 43 byte(s) of data from znode /hbase/meta-region-server; data=asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:14:50,651 DEBUG [Thread-200-EventThread] zookeeper.ZooKeeperWatcher(274): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/region-in-transition/567d8ef20f90f3c1b8a889d22a1c3948 2013-04-07 12:14:50,650 INFO [hbase-am-zkevent-worker-pool-20-thread-17] master.RegionStates(264): Region {NAME => 'testLogSplittingAfterMasterRecoveryDueToZKExpiry,3,1365336889784.4e9a3d63237d29e6bf55bf2360834cbe.', STARTKEY => '3', ENDKEY => '4', ENCODED => 4e9a3d63237d29e6bf55bf2360834cbe,} transitioned from {testLogSplittingAfterMasterRecoveryDueToZKExpiry,3,1365336889784.4e9a3d63237d29e6bf55bf2360834cbe. state=OPENING, ts=1365336890613, server=asf001.sp2.ygridcore.net,60884,1365336878389} to {testLogSplittingAfterMasterRecoveryDueToZKExpiry,3,1365336889784.4e9a3d63237d29e6bf55bf2360834cbe. state=OPENING, ts=1365336890650, server=asf001.sp2.ygridcore.net,60884,1365336878389} 2013-04-07 12:14:50,651 DEBUG [RS_OPEN_REGION-asf001.sp2.ygridcore.net,60884,1365336878389-1] zookeeper.ZKAssign(784): regionserver:60884-0x13de46b5f8b0001 Successfully transitioned node 5403a557a6168e5f3c5141eb494f6088 from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENING 2013-04-07 12:14:50,651 DEBUG [PostOpenDeployTasks:4e9a3d63237d29e6bf55bf2360834cbe] zookeeper.ZKUtil(1680): hconnection-0x43ad4a-0x13de46b5f8b0005 Retrieved 43 byte(s) of data from znode /hbase/meta-region-server; data=asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:14:50,651 INFO [PostOpenDeployTasks:5403a557a6168e5f3c5141eb494f6088] regionserver.HRegionServer(1563): Post open deploy tasks for region=testLogSplittingAfterMasterRecoveryDueToZKExpiry,5,1365336889784.5403a557a6168e5f3c5141eb494f6088. 2013-04-07 12:14:50,652 DEBUG [Thread-200-EventThread] zookeeper.ZooKeeperWatcher(274): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/region-in-transition/5403a557a6168e5f3c5141eb494f6088 2013-04-07 12:14:50,653 INFO [hbase-am-zkevent-worker-pool-2-thread-9] master.RegionStates(264): Region {NAME => 'testLogSplittingAfterMasterRecoveryDueToZKExpiry,,1365336889784.fb4182aef4ce07f011871ae0a083aee0.', STARTKEY => '', ENDKEY => '1', ENCODED => fb4182aef4ce07f011871ae0a083aee0,} transitioned from {testLogSplittingAfterMasterRecoveryDueToZKExpiry,,1365336889784.fb4182aef4ce07f011871ae0a083aee0. state=OFFLINE, ts=1365336890653, server=null} to {testLogSplittingAfterMasterRecoveryDueToZKExpiry,,1365336889784.fb4182aef4ce07f011871ae0a083aee0. state=OPENING, ts=1365336890653, server=asf001.sp2.ygridcore.net,60884,1365336878389} 2013-04-07 12:14:50,653 DEBUG [hbase-am-zkevent-worker-pool-2-thread-13] master.AssignmentManager(740): Handling transition=RS_ZK_REGION_OPENING, server=asf001.sp2.ygridcore.net,60884,1365336878389, region=4e9a3d63237d29e6bf55bf2360834cbe, current state from region state map ={testLogSplittingAfterMasterRecoveryDueToZKExpiry,3,1365336889784.4e9a3d63237d29e6bf55bf2360834cbe. state=OPENING, ts=1365336890625, server=asf001.sp2.ygridcore.net,60884,1365336878389} 2013-04-07 12:14:50,653 INFO [hbase-am-zkevent-worker-pool-2-thread-13] master.RegionStates(264): Region {NAME => 'testLogSplittingAfterMasterRecoveryDueToZKExpiry,3,1365336889784.4e9a3d63237d29e6bf55bf2360834cbe.', STARTKEY => '3', ENDKEY => '4', ENCODED => 4e9a3d63237d29e6bf55bf2360834cbe,} transitioned from {testLogSplittingAfterMasterRecoveryDueToZKExpiry,3,1365336889784.4e9a3d63237d29e6bf55bf2360834cbe. state=OPENING, ts=1365336890625, server=asf001.sp2.ygridcore.net,60884,1365336878389} to {testLogSplittingAfterMasterRecoveryDueToZKExpiry,3,1365336889784.4e9a3d63237d29e6bf55bf2360834cbe. state=OPENING, ts=1365336890653, server=asf001.sp2.ygridcore.net,60884,1365336878389} 2013-04-07 12:14:50,653 INFO [hbase-am-zkevent-worker-pool-2-thread-12] master.RegionStates(264): Region {NAME => 'testLogSplittingAfterMasterRecoveryDueToZKExpiry,1,1365336889784.43207168ac66dce85cf2a4f916007450.', STARTKEY => '1', ENDKEY => '2', ENCODED => 43207168ac66dce85cf2a4f916007450,} transitioned from {testLogSplittingAfterMasterRecoveryDueToZKExpiry,1,1365336889784.43207168ac66dce85cf2a4f916007450. state=OPENING, ts=1365336890608, server=asf001.sp2.ygridcore.net,43802,1365336889418} to {testLogSplittingAfterMasterRecoveryDueToZKExpiry,1,1365336889784.43207168ac66dce85cf2a4f916007450. state=OPENING, ts=1365336890653, server=asf001.sp2.ygridcore.net,43802,1365336889418} 2013-04-07 12:14:50,654 DEBUG [PostOpenDeployTasks:5403a557a6168e5f3c5141eb494f6088] zookeeper.ZKUtil(1680): hconnection-0x43ad4a-0x13de46b5f8b0005 Retrieved 43 byte(s) of data from znode /hbase/meta-region-server; data=asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:14:50,658 DEBUG [hbase-am-zkevent-worker-pool-20-thread-18] zookeeper.ZKUtil(1680): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Retrieved 152 byte(s) of data from znode /hbase/region-in-transition/567d8ef20f90f3c1b8a889d22a1c3948 and set watcher; region=testLogSplittingAfterMasterRecoveryDueToZKExpiry,4,1365336889784.567d8ef20f90f3c1b8a889d22a1c3948., state=RS_ZK_REGION_OPENED, servername=asf001.sp2.ygridcore.net,43802,1365336889418, createTime=1365336890644, payload.length=0 2013-04-07 12:14:50,658 DEBUG [hbase-am-zkevent-worker-pool-20-thread-18] master.AssignmentManager(740): Handling transition=RS_ZK_REGION_OPENED, server=asf001.sp2.ygridcore.net,43802,1365336889418, region=567d8ef20f90f3c1b8a889d22a1c3948, current state from region state map ={testLogSplittingAfterMasterRecoveryDueToZKExpiry,4,1365336889784.567d8ef20f90f3c1b8a889d22a1c3948. state=OPENING, ts=1365336890624, server=asf001.sp2.ygridcore.net,43802,1365336889418} 2013-04-07 12:14:50,658 INFO [hbase-am-zkevent-worker-pool-20-thread-18] master.RegionStates(264): Region {NAME => 'testLogSplittingAfterMasterRecoveryDueToZKExpiry,4,1365336889784.567d8ef20f90f3c1b8a889d22a1c3948.', STARTKEY => '4', ENDKEY => '5', ENCODED => 567d8ef20f90f3c1b8a889d22a1c3948,} transitioned from {testLogSplittingAfterMasterRecoveryDueToZKExpiry,4,1365336889784.567d8ef20f90f3c1b8a889d22a1c3948. state=OPENING, ts=1365336890624, server=asf001.sp2.ygridcore.net,43802,1365336889418} to {testLogSplittingAfterMasterRecoveryDueToZKExpiry,4,1365336889784.567d8ef20f90f3c1b8a889d22a1c3948. state=OPEN, ts=1365336890658, server=asf001.sp2.ygridcore.net,43802,1365336889418} 2013-04-07 12:14:50,659 DEBUG [MASTER_OPEN_REGION-asf001.sp2.ygridcore.net,33796,1365336878036-0] handler.OpenedRegionHandler(146): Handling OPENED event for testLogSplittingAfterMasterRecoveryDueToZKExpiry,4,1365336889784.567d8ef20f90f3c1b8a889d22a1c3948. from asf001.sp2.ygridcore.net,43802,1365336889418; deleting unassigned node 2013-04-07 12:14:50,659 DEBUG [MASTER_OPEN_REGION-asf001.sp2.ygridcore.net,33796,1365336878036-0] zookeeper.ZKAssign(405): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Deleting existing unassigned node for 567d8ef20f90f3c1b8a889d22a1c3948 that is in expected state RS_ZK_REGION_OPENED 2013-04-07 12:14:50,659 DEBUG [hbase-am-zkevent-worker-pool-2-thread-14] zookeeper.ZKUtil(1680): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Retrieved 152 byte(s) of data from znode /hbase/region-in-transition/567d8ef20f90f3c1b8a889d22a1c3948 and set watcher; region=testLogSplittingAfterMasterRecoveryDueToZKExpiry,4,1365336889784.567d8ef20f90f3c1b8a889d22a1c3948., state=RS_ZK_REGION_OPENED, servername=asf001.sp2.ygridcore.net,43802,1365336889418, createTime=1365336890644, payload.length=0 2013-04-07 12:14:50,664 DEBUG [hbase-am-zkevent-worker-pool-20-thread-19] zookeeper.ZKUtil(1680): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Retrieved 152 byte(s) of data from znode /hbase/region-in-transition/5403a557a6168e5f3c5141eb494f6088 and set watcher; region=testLogSplittingAfterMasterRecoveryDueToZKExpiry,5,1365336889784.5403a557a6168e5f3c5141eb494f6088., state=RS_ZK_REGION_OPENING, servername=asf001.sp2.ygridcore.net,60884,1365336878389, createTime=1365336890646, payload.length=0 2013-04-07 12:14:50,664 DEBUG [hbase-am-zkevent-worker-pool-20-thread-19] master.AssignmentManager(740): Handling transition=RS_ZK_REGION_OPENING, server=asf001.sp2.ygridcore.net,60884,1365336878389, region=5403a557a6168e5f3c5141eb494f6088, current state from region state map ={testLogSplittingAfterMasterRecoveryDueToZKExpiry,5,1365336889784.5403a557a6168e5f3c5141eb494f6088. state=OPENING, ts=1365336890623, server=asf001.sp2.ygridcore.net,60884,1365336878389} 2013-04-07 12:14:50,664 INFO [hbase-am-zkevent-worker-pool-20-thread-19] master.RegionStates(264): Region {NAME => 'testLogSplittingAfterMasterRecoveryDueToZKExpiry,5,1365336889784.5403a557a6168e5f3c5141eb494f6088.', STARTKEY => '5', ENDKEY => '', ENCODED => 5403a557a6168e5f3c5141eb494f6088,} transitioned from {testLogSplittingAfterMasterRecoveryDueToZKExpiry,5,1365336889784.5403a557a6168e5f3c5141eb494f6088. state=OPENING, ts=1365336890623, server=asf001.sp2.ygridcore.net,60884,1365336878389} to {testLogSplittingAfterMasterRecoveryDueToZKExpiry,5,1365336889784.5403a557a6168e5f3c5141eb494f6088. state=OPENING, ts=1365336890664, server=asf001.sp2.ygridcore.net,60884,1365336878389} 2013-04-07 12:14:50,665 DEBUG [hbase-am-zkevent-worker-pool-2-thread-15] zookeeper.ZKUtil(1680): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Retrieved 152 byte(s) of data from znode /hbase/region-in-transition/5403a557a6168e5f3c5141eb494f6088 and set watcher; region=testLogSplittingAfterMasterRecoveryDueToZKExpiry,5,1365336889784.5403a557a6168e5f3c5141eb494f6088., state=RS_ZK_REGION_OPENING, servername=asf001.sp2.ygridcore.net,60884,1365336878389, createTime=1365336890646, payload.length=0 2013-04-07 12:14:50,665 DEBUG [hbase-am-zkevent-worker-pool-2-thread-15] master.AssignmentManager(740): Handling transition=RS_ZK_REGION_OPENING, server=asf001.sp2.ygridcore.net,60884,1365336878389, region=5403a557a6168e5f3c5141eb494f6088, current state from region state map ={testLogSplittingAfterMasterRecoveryDueToZKExpiry,5,1365336889784.5403a557a6168e5f3c5141eb494f6088. state=OPENING, ts=1365336890640, server=asf001.sp2.ygridcore.net,60884,1365336878389} 2013-04-07 12:14:50,665 INFO [hbase-am-zkevent-worker-pool-2-thread-15] master.RegionStates(264): Region {NAME => 'testLogSplittingAfterMasterRecoveryDueToZKExpiry,5,1365336889784.5403a557a6168e5f3c5141eb494f6088.', STARTKEY => '5', ENDKEY => '', ENCODED => 5403a557a6168e5f3c5141eb494f6088,} transitioned from {testLogSplittingAfterMasterRecoveryDueToZKExpiry,5,1365336889784.5403a557a6168e5f3c5141eb494f6088. state=OPENING, ts=1365336890640, server=asf001.sp2.ygridcore.net,60884,1365336878389} to {testLogSplittingAfterMasterRecoveryDueToZKExpiry,5,1365336889784.5403a557a6168e5f3c5141eb494f6088. state=OPENING, ts=1365336890665, server=asf001.sp2.ygridcore.net,60884,1365336878389} 2013-04-07 12:14:50,666 DEBUG [Thread-200-EventThread] zookeeper.ZooKeeperWatcher(274): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/region-in-transition/43207168ac66dce85cf2a4f916007450 2013-04-07 12:14:50,671 INFO [PostOpenDeployTasks:5403a557a6168e5f3c5141eb494f6088] catalog.MetaEditor(432): Updated row testLogSplittingAfterMasterRecoveryDueToZKExpiry,5,1365336889784.5403a557a6168e5f3c5141eb494f6088. with server=asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:14:50,671 INFO [PostOpenDeployTasks:5403a557a6168e5f3c5141eb494f6088] regionserver.HRegionServer(1584): Done with post open deploy task for region=testLogSplittingAfterMasterRecoveryDueToZKExpiry,5,1365336889784.5403a557a6168e5f3c5141eb494f6088. 2013-04-07 12:14:50,671 DEBUG [RS_OPEN_REGION-asf001.sp2.ygridcore.net,60884,1365336878389-1] zookeeper.ZKAssign(721): regionserver:60884-0x13de46b5f8b0001 Attempting to transition node 5403a557a6168e5f3c5141eb494f6088 from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENED 2013-04-07 12:14:50,673 DEBUG [Thread-200-EventThread] zookeeper.ZooKeeperWatcher(274): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/region-in-transition/55884b67b9203a71bdf8c0e3de70452d 2013-04-07 12:14:50,674 DEBUG [RS_OPEN_REGION-asf001.sp2.ygridcore.net,60884,1365336878389-0] zookeeper.ZKAssign(784): regionserver:60884-0x13de46b5f8b0001 Successfully transitioned node 4e9a3d63237d29e6bf55bf2360834cbe from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENED 2013-04-07 12:14:50,674 DEBUG [RS_OPEN_REGION-asf001.sp2.ygridcore.net,60884,1365336878389-0] handler.OpenRegionHandler(361): region transitioned to opened in zookeeper: {NAME => 'testLogSplittingAfterMasterRecoveryDueToZKExpiry,3,1365336889784.4e9a3d63237d29e6bf55bf2360834cbe.', STARTKEY => '3', ENDKEY => '4', ENCODED => 4e9a3d63237d29e6bf55bf2360834cbe,}, server: asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:14:50,675 DEBUG [RS_OPEN_REGION-asf001.sp2.ygridcore.net,60884,1365336878389-0] handler.OpenRegionHandler(168): Opened testLogSplittingAfterMasterRecoveryDueToZKExpiry,3,1365336889784.4e9a3d63237d29e6bf55bf2360834cbe. on server:asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:14:50,694 DEBUG [Thread-200-EventThread] zookeeper.ZooKeeperWatcher(274): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/region-in-transition/4e9a3d63237d29e6bf55bf2360834cbe 2013-04-07 12:14:50,703 DEBUG [hbase-am-zkevent-worker-pool-20-thread-20] zookeeper.ZKUtil(1680): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Retrieved 152 byte(s) of data from znode /hbase/region-in-transition/43207168ac66dce85cf2a4f916007450 and set watcher; region=testLogSplittingAfterMasterRecoveryDueToZKExpiry,1,1365336889784.43207168ac66dce85cf2a4f916007450., state=RS_ZK_REGION_OPENED, servername=asf001.sp2.ygridcore.net,43802,1365336889418, createTime=1365336890654, payload.length=0 2013-04-07 12:14:50,703 INFO [hbase-am-zkevent-worker-pool-2-thread-16] master.RegionStates(264): Region {NAME => 'testLogSplittingAfterMasterRecoveryDueToZKExpiry,1,1365336889784.43207168ac66dce85cf2a4f916007450.', STARTKEY => '1', ENDKEY => '2', ENCODED => 43207168ac66dce85cf2a4f916007450,} transitioned from {testLogSplittingAfterMasterRecoveryDueToZKExpiry,1,1365336889784.43207168ac66dce85cf2a4f916007450. state=OPENING, ts=1365336890653, server=asf001.sp2.ygridcore.net,43802,1365336889418} to {testLogSplittingAfterMasterRecoveryDueToZKExpiry,1,1365336889784.43207168ac66dce85cf2a4f916007450. state=OPEN, ts=1365336890703, server=asf001.sp2.ygridcore.net,43802,1365336889418} 2013-04-07 12:14:50,704 DEBUG [hbase-am-zkevent-worker-pool-20-thread-20] master.AssignmentManager(740): Handling transition=RS_ZK_REGION_OPENED, server=asf001.sp2.ygridcore.net,43802,1365336889418, region=43207168ac66dce85cf2a4f916007450, current state from region state map ={testLogSplittingAfterMasterRecoveryDueToZKExpiry,1,1365336889784.43207168ac66dce85cf2a4f916007450. state=OPENING, ts=1365336890638, server=asf001.sp2.ygridcore.net,43802,1365336889418} 2013-04-07 12:14:50,704 DEBUG [MASTER_OPEN_REGION-asf001.sp2.ygridcore.net,33796,1365336878036-2] handler.OpenedRegionHandler(146): Handling OPENED event for testLogSplittingAfterMasterRecoveryDueToZKExpiry,1,1365336889784.43207168ac66dce85cf2a4f916007450. from asf001.sp2.ygridcore.net,43802,1365336889418; deleting unassigned node 2013-04-07 12:14:50,704 DEBUG [MASTER_OPEN_REGION-asf001.sp2.ygridcore.net,33796,1365336878036-2] zookeeper.ZKAssign(405): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Deleting existing unassigned node for 43207168ac66dce85cf2a4f916007450 that is in expected state RS_ZK_REGION_OPENED 2013-04-07 12:14:50,704 DEBUG [Thread-200-EventThread] zookeeper.ZooKeeperWatcher(274): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/region-in-transition/5403a557a6168e5f3c5141eb494f6088 2013-04-07 12:14:50,704 INFO [hbase-am-zkevent-worker-pool-20-thread-20] master.RegionStates(264): Region {NAME => 'testLogSplittingAfterMasterRecoveryDueToZKExpiry,1,1365336889784.43207168ac66dce85cf2a4f916007450.', STARTKEY => '1', ENDKEY => '2', ENCODED => 43207168ac66dce85cf2a4f916007450,} transitioned from {testLogSplittingAfterMasterRecoveryDueToZKExpiry,1,1365336889784.43207168ac66dce85cf2a4f916007450. state=OPENING, ts=1365336890638, server=asf001.sp2.ygridcore.net,43802,1365336889418} to {testLogSplittingAfterMasterRecoveryDueToZKExpiry,1,1365336889784.43207168ac66dce85cf2a4f916007450. state=OPEN, ts=1365336890704, server=asf001.sp2.ygridcore.net,43802,1365336889418} 2013-04-07 12:14:50,704 DEBUG [MASTER_OPEN_REGION-asf001.sp2.ygridcore.net,33796,1365336878036-3] handler.OpenedRegionHandler(146): Handling OPENED event for testLogSplittingAfterMasterRecoveryDueToZKExpiry,1,1365336889784.43207168ac66dce85cf2a4f916007450. from asf001.sp2.ygridcore.net,43802,1365336889418; deleting unassigned node 2013-04-07 12:14:50,704 DEBUG [MASTER_OPEN_REGION-asf001.sp2.ygridcore.net,33796,1365336878036-3] zookeeper.ZKAssign(405): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Deleting existing unassigned node for 43207168ac66dce85cf2a4f916007450 that is in expected state RS_ZK_REGION_OPENED 2013-04-07 12:14:50,705 DEBUG [Thread-200-EventThread] zookeeper.ZooKeeperWatcher(274): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/region-in-transition/fb4182aef4ce07f011871ae0a083aee0 2013-04-07 12:14:50,706 INFO [PostOpenDeployTasks:fb4182aef4ce07f011871ae0a083aee0] catalog.MetaEditor(432): Updated row testLogSplittingAfterMasterRecoveryDueToZKExpiry,,1365336889784.fb4182aef4ce07f011871ae0a083aee0. with server=asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:14:50,706 INFO [PostOpenDeployTasks:fb4182aef4ce07f011871ae0a083aee0] regionserver.HRegionServer(1584): Done with post open deploy task for region=testLogSplittingAfterMasterRecoveryDueToZKExpiry,,1365336889784.fb4182aef4ce07f011871ae0a083aee0. 2013-04-07 12:14:50,707 DEBUG [hbase-am-zkevent-worker-pool-20-thread-1] zookeeper.ZKUtil(1680): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Retrieved 152 byte(s) of data from znode /hbase/region-in-transition/55884b67b9203a71bdf8c0e3de70452d and set watcher; region=testLogSplittingAfterMasterRecoveryDueToZKExpiry,2,1365336889784.55884b67b9203a71bdf8c0e3de70452d., state=RS_ZK_REGION_OPENED, servername=asf001.sp2.ygridcore.net,43802,1365336889418, createTime=1365336890696, payload.length=0 2013-04-07 12:14:50,707 DEBUG [RS_OPEN_REGION-asf001.sp2.ygridcore.net,60884,1365336878389-2] zookeeper.ZKAssign(721): regionserver:60884-0x13de46b5f8b0001 Attempting to transition node fb4182aef4ce07f011871ae0a083aee0 from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENED 2013-04-07 12:14:50,707 DEBUG [hbase-am-zkevent-worker-pool-20-thread-1] master.AssignmentManager(740): Handling transition=RS_ZK_REGION_OPENED, server=asf001.sp2.ygridcore.net,43802,1365336889418, region=55884b67b9203a71bdf8c0e3de70452d, current state from region state map ={testLogSplittingAfterMasterRecoveryDueToZKExpiry,2,1365336889784.55884b67b9203a71bdf8c0e3de70452d. state=OPENING, ts=1365336890573, server=asf001.sp2.ygridcore.net,43802,1365336889418} 2013-04-07 12:14:50,708 INFO [hbase-am-zkevent-worker-pool-20-thread-1] master.RegionStates(264): Region {NAME => 'testLogSplittingAfterMasterRecoveryDueToZKExpiry,2,1365336889784.55884b67b9203a71bdf8c0e3de70452d.', STARTKEY => '2', ENDKEY => '3', ENCODED => 55884b67b9203a71bdf8c0e3de70452d,} transitioned from {testLogSplittingAfterMasterRecoveryDueToZKExpiry,2,1365336889784.55884b67b9203a71bdf8c0e3de70452d. state=OPENING, ts=1365336890573, server=asf001.sp2.ygridcore.net,43802,1365336889418} to {testLogSplittingAfterMasterRecoveryDueToZKExpiry,2,1365336889784.55884b67b9203a71bdf8c0e3de70452d. state=OPEN, ts=1365336890708, server=asf001.sp2.ygridcore.net,43802,1365336889418} 2013-04-07 12:14:50,708 DEBUG [MASTER_OPEN_REGION-asf001.sp2.ygridcore.net,33796,1365336878036-4] handler.OpenedRegionHandler(146): Handling OPENED event for testLogSplittingAfterMasterRecoveryDueToZKExpiry,2,1365336889784.55884b67b9203a71bdf8c0e3de70452d. from asf001.sp2.ygridcore.net,43802,1365336889418; deleting unassigned node 2013-04-07 12:14:50,708 DEBUG [MASTER_OPEN_REGION-asf001.sp2.ygridcore.net,33796,1365336878036-4] zookeeper.ZKAssign(405): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Deleting existing unassigned node for 55884b67b9203a71bdf8c0e3de70452d that is in expected state RS_ZK_REGION_OPENED 2013-04-07 12:14:50,708 DEBUG [hbase-am-zkevent-worker-pool-2-thread-17] zookeeper.ZKUtil(1680): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Retrieved 152 byte(s) of data from znode /hbase/region-in-transition/55884b67b9203a71bdf8c0e3de70452d and set watcher; region=testLogSplittingAfterMasterRecoveryDueToZKExpiry,2,1365336889784.55884b67b9203a71bdf8c0e3de70452d., state=RS_ZK_REGION_OPENED, servername=asf001.sp2.ygridcore.net,43802,1365336889418, createTime=1365336890696, payload.length=0 2013-04-07 12:14:50,708 DEBUG [hbase-am-zkevent-worker-pool-2-thread-17] master.AssignmentManager(740): Handling transition=RS_ZK_REGION_OPENED, server=asf001.sp2.ygridcore.net,43802,1365336889418, region=55884b67b9203a71bdf8c0e3de70452d, current state from region state map ={testLogSplittingAfterMasterRecoveryDueToZKExpiry,2,1365336889784.55884b67b9203a71bdf8c0e3de70452d. state=OPENING, ts=1365336890587, server=asf001.sp2.ygridcore.net,43802,1365336889418} 2013-04-07 12:14:50,709 INFO [hbase-am-zkevent-worker-pool-2-thread-17] master.RegionStates(264): Region {NAME => 'testLogSplittingAfterMasterRecoveryDueToZKExpiry,2,1365336889784.55884b67b9203a71bdf8c0e3de70452d.', STARTKEY => '2', ENDKEY => '3', ENCODED => 55884b67b9203a71bdf8c0e3de70452d,} transitioned from {testLogSplittingAfterMasterRecoveryDueToZKExpiry,2,1365336889784.55884b67b9203a71bdf8c0e3de70452d. state=OPENING, ts=1365336890587, server=asf001.sp2.ygridcore.net,43802,1365336889418} to {testLogSplittingAfterMasterRecoveryDueToZKExpiry,2,1365336889784.55884b67b9203a71bdf8c0e3de70452d. state=OPEN, ts=1365336890709, server=asf001.sp2.ygridcore.net,43802,1365336889418} 2013-04-07 12:14:50,709 DEBUG [RS_OPEN_REGION-asf001.sp2.ygridcore.net,60884,1365336878389-2] zookeeper.ZKUtil(1680): regionserver:60884-0x13de46b5f8b0001 Retrieved 151 byte(s) of data from znode /hbase/region-in-transition/fb4182aef4ce07f011871ae0a083aee0; data=region=testLogSplittingAfterMasterRecoveryDueToZKExpiry,,1365336889784.fb4182aef4ce07f011871ae0a083aee0., state=RS_ZK_REGION_OPENING, servername=asf001.sp2.ygridcore.net,60884,1365336878389, createTime=1365336890693, payload.length=0 2013-04-07 12:14:50,710 DEBUG [hbase-am-zkevent-worker-pool-20-thread-2] zookeeper.ZKUtil(1680): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Retrieved 152 byte(s) of data from znode /hbase/region-in-transition/4e9a3d63237d29e6bf55bf2360834cbe and set watcher; region=testLogSplittingAfterMasterRecoveryDueToZKExpiry,3,1365336889784.4e9a3d63237d29e6bf55bf2360834cbe., state=RS_ZK_REGION_OPENED, servername=asf001.sp2.ygridcore.net,60884,1365336878389, createTime=1365336890662, payload.length=0 2013-04-07 12:14:50,710 DEBUG [hbase-am-zkevent-worker-pool-20-thread-2] master.AssignmentManager(740): Handling transition=RS_ZK_REGION_OPENED, server=asf001.sp2.ygridcore.net,60884,1365336878389, region=4e9a3d63237d29e6bf55bf2360834cbe, current state from region state map ={testLogSplittingAfterMasterRecoveryDueToZKExpiry,3,1365336889784.4e9a3d63237d29e6bf55bf2360834cbe. state=OPENING, ts=1365336890650, server=asf001.sp2.ygridcore.net,60884,1365336878389} 2013-04-07 12:14:50,711 INFO [hbase-am-zkevent-worker-pool-20-thread-2] master.RegionStates(264): Region {NAME => 'testLogSplittingAfterMasterRecoveryDueToZKExpiry,3,1365336889784.4e9a3d63237d29e6bf55bf2360834cbe.', STARTKEY => '3', ENDKEY => '4', ENCODED => 4e9a3d63237d29e6bf55bf2360834cbe,} transitioned from {testLogSplittingAfterMasterRecoveryDueToZKExpiry,3,1365336889784.4e9a3d63237d29e6bf55bf2360834cbe. state=OPENING, ts=1365336890650, server=asf001.sp2.ygridcore.net,60884,1365336878389} to {testLogSplittingAfterMasterRecoveryDueToZKExpiry,3,1365336889784.4e9a3d63237d29e6bf55bf2360834cbe. state=OPEN, ts=1365336890711, server=asf001.sp2.ygridcore.net,60884,1365336878389} 2013-04-07 12:14:50,711 DEBUG [hbase-am-zkevent-worker-pool-2-thread-18] zookeeper.ZKUtil(1680): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Retrieved 152 byte(s) of data from znode /hbase/region-in-transition/4e9a3d63237d29e6bf55bf2360834cbe and set watcher; region=testLogSplittingAfterMasterRecoveryDueToZKExpiry,3,1365336889784.4e9a3d63237d29e6bf55bf2360834cbe., state=RS_ZK_REGION_OPENED, servername=asf001.sp2.ygridcore.net,60884,1365336878389, createTime=1365336890662, payload.length=0 2013-04-07 12:14:50,712 DEBUG [hbase-am-zkevent-worker-pool-2-thread-18] master.AssignmentManager(740): Handling transition=RS_ZK_REGION_OPENED, server=asf001.sp2.ygridcore.net,60884,1365336878389, region=4e9a3d63237d29e6bf55bf2360834cbe, current state from region state map ={testLogSplittingAfterMasterRecoveryDueToZKExpiry,3,1365336889784.4e9a3d63237d29e6bf55bf2360834cbe. state=OPENING, ts=1365336890653, server=asf001.sp2.ygridcore.net,60884,1365336878389} 2013-04-07 12:14:50,712 INFO [hbase-am-zkevent-worker-pool-2-thread-18] master.RegionStates(264): Region {NAME => 'testLogSplittingAfterMasterRecoveryDueToZKExpiry,3,1365336889784.4e9a3d63237d29e6bf55bf2360834cbe.', STARTKEY => '3', ENDKEY => '4', ENCODED => 4e9a3d63237d29e6bf55bf2360834cbe,} transitioned from {testLogSplittingAfterMasterRecoveryDueToZKExpiry,3,1365336889784.4e9a3d63237d29e6bf55bf2360834cbe. state=OPENING, ts=1365336890653, server=asf001.sp2.ygridcore.net,60884,1365336878389} to {testLogSplittingAfterMasterRecoveryDueToZKExpiry,3,1365336889784.4e9a3d63237d29e6bf55bf2360834cbe. state=OPEN, ts=1365336890712, server=asf001.sp2.ygridcore.net,60884,1365336878389} 2013-04-07 12:14:50,712 DEBUG [RS_OPEN_REGION-asf001.sp2.ygridcore.net,60884,1365336878389-2] zookeeper.ZKAssign(784): regionserver:60884-0x13de46b5f8b0001 Successfully transitioned node fb4182aef4ce07f011871ae0a083aee0 from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENED 2013-04-07 12:14:50,712 DEBUG [RS_OPEN_REGION-asf001.sp2.ygridcore.net,60884,1365336878389-2] handler.OpenRegionHandler(361): region transitioned to opened in zookeeper: {NAME => 'testLogSplittingAfterMasterRecoveryDueToZKExpiry,,1365336889784.fb4182aef4ce07f011871ae0a083aee0.', STARTKEY => '', ENDKEY => '1', ENCODED => fb4182aef4ce07f011871ae0a083aee0,}, server: asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:14:50,712 DEBUG [RS_OPEN_REGION-asf001.sp2.ygridcore.net,60884,1365336878389-2] handler.OpenRegionHandler(168): Opened testLogSplittingAfterMasterRecoveryDueToZKExpiry,,1365336889784.fb4182aef4ce07f011871ae0a083aee0. on server:asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:14:50,713 DEBUG [Thread-200-EventThread] zookeeper.ZooKeeperWatcher(274): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/region-in-transition/567d8ef20f90f3c1b8a889d22a1c3948 2013-04-07 12:14:50,713 DEBUG [hbase-am-zkevent-worker-pool-20-thread-5] master.AssignmentManager$4(1159): The znode of region testLogSplittingAfterMasterRecoveryDueToZKExpiry,4,1365336889784.567d8ef20f90f3c1b8a889d22a1c3948. has been deleted. 2013-04-07 12:14:50,713 DEBUG [hbase-am-zkevent-worker-pool-2-thread-1] master.AssignmentManager$4(1159): The znode of region testLogSplittingAfterMasterRecoveryDueToZKExpiry,4,1365336889784.567d8ef20f90f3c1b8a889d22a1c3948. has been deleted. 2013-04-07 12:14:50,713 DEBUG [Thread-200-EventThread] zookeeper.ZooKeeperWatcher(274): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/region-in-transition 2013-04-07 12:14:50,713 INFO [hbase-am-zkevent-worker-pool-2-thread-1] master.RegionStates(264): Region {NAME => 'testLogSplittingAfterMasterRecoveryDueToZKExpiry,4,1365336889784.567d8ef20f90f3c1b8a889d22a1c3948.', STARTKEY => '4', ENDKEY => '5', ENCODED => 567d8ef20f90f3c1b8a889d22a1c3948,} transitioned from {testLogSplittingAfterMasterRecoveryDueToZKExpiry,4,1365336889784.567d8ef20f90f3c1b8a889d22a1c3948. state=OPEN, ts=1365336890660, server=asf001.sp2.ygridcore.net,43802,1365336889418} to {testLogSplittingAfterMasterRecoveryDueToZKExpiry,4,1365336889784.567d8ef20f90f3c1b8a889d22a1c3948. state=OPEN, ts=1365336890713, server=asf001.sp2.ygridcore.net,43802,1365336889418} 2013-04-07 12:14:50,713 INFO [hbase-am-zkevent-worker-pool-20-thread-5] master.RegionStates(264): Region {NAME => 'testLogSplittingAfterMasterRecoveryDueToZKExpiry,4,1365336889784.567d8ef20f90f3c1b8a889d22a1c3948.', STARTKEY => '4', ENDKEY => '5', ENCODED => 567d8ef20f90f3c1b8a889d22a1c3948,} transitioned from {testLogSplittingAfterMasterRecoveryDueToZKExpiry,4,1365336889784.567d8ef20f90f3c1b8a889d22a1c3948. state=OPEN, ts=1365336890658, server=asf001.sp2.ygridcore.net,43802,1365336889418} to {testLogSplittingAfterMasterRecoveryDueToZKExpiry,4,1365336889784.567d8ef20f90f3c1b8a889d22a1c3948. state=OPEN, ts=1365336890713, server=asf001.sp2.ygridcore.net,43802,1365336889418} 2013-04-07 12:14:50,713 INFO [hbase-am-zkevent-worker-pool-2-thread-1] master.RegionStates(299): Onlined region {NAME => 'testLogSplittingAfterMasterRecoveryDueToZKExpiry,4,1365336889784.567d8ef20f90f3c1b8a889d22a1c3948.', STARTKEY => '4', ENDKEY => '5', ENCODED => 567d8ef20f90f3c1b8a889d22a1c3948,} on asf001.sp2.ygridcore.net,43802,1365336889418 2013-04-07 12:14:50,714 INFO [hbase-am-zkevent-worker-pool-20-thread-5] master.RegionStates(299): Onlined region {NAME => 'testLogSplittingAfterMasterRecoveryDueToZKExpiry,4,1365336889784.567d8ef20f90f3c1b8a889d22a1c3948.', STARTKEY => '4', ENDKEY => '5', ENCODED => 567d8ef20f90f3c1b8a889d22a1c3948,} on asf001.sp2.ygridcore.net,43802,1365336889418 2013-04-07 12:14:50,714 INFO [hbase-am-zkevent-worker-pool-2-thread-1] master.AssignmentManager$4(1164): The master has opened the region testLogSplittingAfterMasterRecoveryDueToZKExpiry,4,1365336889784.567d8ef20f90f3c1b8a889d22a1c3948. that was online on asf001.sp2.ygridcore.net,43802,1365336889418 2013-04-07 12:14:50,714 DEBUG [MASTER_OPEN_REGION-asf001.sp2.ygridcore.net,33796,1365336878036-0] zookeeper.ZKAssign(434): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Successfully deleted unassigned node for region 567d8ef20f90f3c1b8a889d22a1c3948 in expected state RS_ZK_REGION_OPENED 2013-04-07 12:14:50,714 INFO [hbase-am-zkevent-worker-pool-20-thread-5] master.AssignmentManager$4(1164): The master has opened the region testLogSplittingAfterMasterRecoveryDueToZKExpiry,4,1365336889784.567d8ef20f90f3c1b8a889d22a1c3948. that was online on asf001.sp2.ygridcore.net,43802,1365336889418 2013-04-07 12:14:50,714 DEBUG [MASTER_OPEN_REGION-asf001.sp2.ygridcore.net,33796,1365336878036-0] handler.OpenedRegionHandler(146): Handling OPENED event for testLogSplittingAfterMasterRecoveryDueToZKExpiry,2,1365336889784.55884b67b9203a71bdf8c0e3de70452d. from asf001.sp2.ygridcore.net,43802,1365336889418; deleting unassigned node 2013-04-07 12:14:50,715 DEBUG [MASTER_OPEN_REGION-asf001.sp2.ygridcore.net,33796,1365336878036-0] zookeeper.ZKAssign(405): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Deleting existing unassigned node for 55884b67b9203a71bdf8c0e3de70452d that is in expected state RS_ZK_REGION_OPENED 2013-04-07 12:14:50,716 WARN [MASTER_OPEN_REGION-asf001.sp2.ygridcore.net,33796,1365336878036-1] zookeeper.RecoverableZooKeeper(161): Node /hbase/region-in-transition/567d8ef20f90f3c1b8a889d22a1c3948 already deleted, retry=false 2013-04-07 12:14:50,716 WARN [MASTER_OPEN_REGION-asf001.sp2.ygridcore.net,33796,1365336878036-1] handler.OpenedRegionHandler(132): The znode of the region testLogSplittingAfterMasterRecoveryDueToZKExpiry,4,1365336889784.567d8ef20f90f3c1b8a889d22a1c3948. would have already been deleted 2013-04-07 12:14:50,716 ERROR [MASTER_OPEN_REGION-asf001.sp2.ygridcore.net,33796,1365336878036-1] handler.OpenedRegionHandler(104): The znode of region testLogSplittingAfterMasterRecoveryDueToZKExpiry,4,1365336889784.567d8ef20f90f3c1b8a889d22a1c3948. could not be deleted. 2013-04-07 12:14:50,716 DEBUG [MASTER_OPEN_REGION-asf001.sp2.ygridcore.net,33796,1365336878036-1] handler.OpenedRegionHandler(146): Handling OPENED event for testLogSplittingAfterMasterRecoveryDueToZKExpiry,3,1365336889784.4e9a3d63237d29e6bf55bf2360834cbe. from asf001.sp2.ygridcore.net,60884,1365336878389; deleting unassigned node 2013-04-07 12:14:50,716 DEBUG [MASTER_OPEN_REGION-asf001.sp2.ygridcore.net,33796,1365336878036-1] zookeeper.ZKAssign(405): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Deleting existing unassigned node for 4e9a3d63237d29e6bf55bf2360834cbe that is in expected state RS_ZK_REGION_OPENED 2013-04-07 12:14:50,717 DEBUG [MASTER_OPEN_REGION-asf001.sp2.ygridcore.net,33796,1365336878036-2] zookeeper.ZKUtil(1680): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Retrieved 152 byte(s) of data from znode /hbase/region-in-transition/43207168ac66dce85cf2a4f916007450; data=region=testLogSplittingAfterMasterRecoveryDueToZKExpiry,1,1365336889784.43207168ac66dce85cf2a4f916007450., state=RS_ZK_REGION_OPENED, servername=asf001.sp2.ygridcore.net,43802,1365336889418, createTime=1365336890654, payload.length=0 2013-04-07 12:14:50,718 DEBUG [hbase-am-zkevent-worker-pool-2-thread-19] zookeeper.ZKUtil(1680): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Retrieved 152 byte(s) of data from znode /hbase/region-in-transition/5403a557a6168e5f3c5141eb494f6088 and set watcher; region=testLogSplittingAfterMasterRecoveryDueToZKExpiry,5,1365336889784.5403a557a6168e5f3c5141eb494f6088., state=RS_ZK_REGION_OPENED, servername=asf001.sp2.ygridcore.net,60884,1365336878389, createTime=1365336890687, payload.length=0 2013-04-07 12:14:50,718 DEBUG [hbase-am-zkevent-worker-pool-2-thread-19] master.AssignmentManager(740): Handling transition=RS_ZK_REGION_OPENED, server=asf001.sp2.ygridcore.net,60884,1365336878389, region=5403a557a6168e5f3c5141eb494f6088, current state from region state map ={testLogSplittingAfterMasterRecoveryDueToZKExpiry,5,1365336889784.5403a557a6168e5f3c5141eb494f6088. state=OPENING, ts=1365336890665, server=asf001.sp2.ygridcore.net,60884,1365336878389} 2013-04-07 12:14:50,718 INFO [hbase-am-zkevent-worker-pool-2-thread-19] master.RegionStates(264): Region {NAME => 'testLogSplittingAfterMasterRecoveryDueToZKExpiry,5,1365336889784.5403a557a6168e5f3c5141eb494f6088.', STARTKEY => '5', ENDKEY => '', ENCODED => 5403a557a6168e5f3c5141eb494f6088,} transitioned from {testLogSplittingAfterMasterRecoveryDueToZKExpiry,5,1365336889784.5403a557a6168e5f3c5141eb494f6088. state=OPENING, ts=1365336890665, server=asf001.sp2.ygridcore.net,60884,1365336878389} to {testLogSplittingAfterMasterRecoveryDueToZKExpiry,5,1365336889784.5403a557a6168e5f3c5141eb494f6088. state=OPEN, ts=1365336890718, server=asf001.sp2.ygridcore.net,60884,1365336878389} 2013-04-07 12:14:50,718 DEBUG [hbase-am-zkevent-worker-pool-20-thread-3] zookeeper.ZKUtil(1680): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Retrieved 152 byte(s) of data from znode /hbase/region-in-transition/5403a557a6168e5f3c5141eb494f6088 and set watcher; region=testLogSplittingAfterMasterRecoveryDueToZKExpiry,5,1365336889784.5403a557a6168e5f3c5141eb494f6088., state=RS_ZK_REGION_OPENED, servername=asf001.sp2.ygridcore.net,60884,1365336878389, createTime=1365336890687, payload.length=0 2013-04-07 12:14:50,718 DEBUG [hbase-am-zkevent-worker-pool-20-thread-3] master.AssignmentManager(740): Handling transition=RS_ZK_REGION_OPENED, server=asf001.sp2.ygridcore.net,60884,1365336878389, region=5403a557a6168e5f3c5141eb494f6088, current state from region state map ={testLogSplittingAfterMasterRecoveryDueToZKExpiry,5,1365336889784.5403a557a6168e5f3c5141eb494f6088. state=OPENING, ts=1365336890664, server=asf001.sp2.ygridcore.net,60884,1365336878389} 2013-04-07 12:14:50,719 INFO [hbase-am-zkevent-worker-pool-20-thread-3] master.RegionStates(264): Region {NAME => 'testLogSplittingAfterMasterRecoveryDueToZKExpiry,5,1365336889784.5403a557a6168e5f3c5141eb494f6088.', STARTKEY => '5', ENDKEY => '', ENCODED => 5403a557a6168e5f3c5141eb494f6088,} transitioned from {testLogSplittingAfterMasterRecoveryDueToZKExpiry,5,1365336889784.5403a557a6168e5f3c5141eb494f6088. state=OPENING, ts=1365336890664, server=asf001.sp2.ygridcore.net,60884,1365336878389} to {testLogSplittingAfterMasterRecoveryDueToZKExpiry,5,1365336889784.5403a557a6168e5f3c5141eb494f6088. state=OPEN, ts=1365336890719, server=asf001.sp2.ygridcore.net,60884,1365336878389} 2013-04-07 12:14:50,719 DEBUG [MASTER_OPEN_REGION-asf001.sp2.ygridcore.net,33796,1365336878036-3] zookeeper.ZKUtil(1680): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Retrieved 152 byte(s) of data from znode /hbase/region-in-transition/43207168ac66dce85cf2a4f916007450; data=region=testLogSplittingAfterMasterRecoveryDueToZKExpiry,1,1365336889784.43207168ac66dce85cf2a4f916007450., state=RS_ZK_REGION_OPENED, servername=asf001.sp2.ygridcore.net,43802,1365336889418, createTime=1365336890654, payload.length=0 2013-04-07 12:14:50,719 DEBUG [hbase-am-zkevent-worker-pool-20-thread-4] zookeeper.ZKUtil(1680): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Retrieved 151 byte(s) of data from znode /hbase/region-in-transition/fb4182aef4ce07f011871ae0a083aee0 and set watcher; region=testLogSplittingAfterMasterRecoveryDueToZKExpiry,,1365336889784.fb4182aef4ce07f011871ae0a083aee0., state=RS_ZK_REGION_OPENING, servername=asf001.sp2.ygridcore.net,60884,1365336878389, createTime=1365336890693, payload.length=0 2013-04-07 12:14:50,719 DEBUG [hbase-am-zkevent-worker-pool-20-thread-4] master.AssignmentManager(740): Handling transition=RS_ZK_REGION_OPENING, server=asf001.sp2.ygridcore.net,60884,1365336878389, region=fb4182aef4ce07f011871ae0a083aee0, current state from region state map ={testLogSplittingAfterMasterRecoveryDueToZKExpiry,,1365336889784.fb4182aef4ce07f011871ae0a083aee0. state=OPENING, ts=1365336890620, server=asf001.sp2.ygridcore.net,60884,1365336878389} 2013-04-07 12:14:50,719 INFO [hbase-am-zkevent-worker-pool-20-thread-4] master.RegionStates(264): Region {NAME => 'testLogSplittingAfterMasterRecoveryDueToZKExpiry,,1365336889784.fb4182aef4ce07f011871ae0a083aee0.', STARTKEY => '', ENDKEY => '1', ENCODED => fb4182aef4ce07f011871ae0a083aee0,} transitioned from {testLogSplittingAfterMasterRecoveryDueToZKExpiry,,1365336889784.fb4182aef4ce07f011871ae0a083aee0. state=OPENING, ts=1365336890620, server=asf001.sp2.ygridcore.net,60884,1365336878389} to {testLogSplittingAfterMasterRecoveryDueToZKExpiry,,1365336889784.fb4182aef4ce07f011871ae0a083aee0. state=OPENING, ts=1365336890719, server=asf001.sp2.ygridcore.net,60884,1365336878389} 2013-04-07 12:14:50,720 DEBUG [Thread-200-EventThread] zookeeper.ZooKeeperWatcher(274): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/region-in-transition/fb4182aef4ce07f011871ae0a083aee0 2013-04-07 12:14:50,721 DEBUG [hbase-am-zkevent-worker-pool-2-thread-20] zookeeper.ZKUtil(1680): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Retrieved 151 byte(s) of data from znode /hbase/region-in-transition/fb4182aef4ce07f011871ae0a083aee0 and set watcher; region=testLogSplittingAfterMasterRecoveryDueToZKExpiry,,1365336889784.fb4182aef4ce07f011871ae0a083aee0., state=RS_ZK_REGION_OPENED, servername=asf001.sp2.ygridcore.net,60884,1365336878389, createTime=1365336890710, payload.length=0 2013-04-07 12:14:50,721 DEBUG [hbase-am-zkevent-worker-pool-2-thread-20] master.AssignmentManager(740): Handling transition=RS_ZK_REGION_OPENED, server=asf001.sp2.ygridcore.net,60884,1365336878389, region=fb4182aef4ce07f011871ae0a083aee0, current state from region state map ={testLogSplittingAfterMasterRecoveryDueToZKExpiry,,1365336889784.fb4182aef4ce07f011871ae0a083aee0. state=OPENING, ts=1365336890653, server=asf001.sp2.ygridcore.net,60884,1365336878389} 2013-04-07 12:14:50,727 INFO [hbase-am-zkevent-worker-pool-2-thread-20] master.RegionStates(264): Region {NAME => 'testLogSplittingAfterMasterRecoveryDueToZKExpiry,,1365336889784.fb4182aef4ce07f011871ae0a083aee0.', STARTKEY => '', ENDKEY => '1', ENCODED => fb4182aef4ce07f011871ae0a083aee0,} transitioned from {testLogSplittingAfterMasterRecoveryDueToZKExpiry,,1365336889784.fb4182aef4ce07f011871ae0a083aee0. state=OPENING, ts=1365336890653, server=asf001.sp2.ygridcore.net,60884,1365336878389} to {testLogSplittingAfterMasterRecoveryDueToZKExpiry,,1365336889784.fb4182aef4ce07f011871ae0a083aee0. state=OPEN, ts=1365336890727, server=asf001.sp2.ygridcore.net,60884,1365336878389} 2013-04-07 12:14:50,729 DEBUG [MASTER_OPEN_REGION-asf001.sp2.ygridcore.net,33796,1365336878036-4] zookeeper.ZKUtil(1680): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Retrieved 152 byte(s) of data from znode /hbase/region-in-transition/55884b67b9203a71bdf8c0e3de70452d; data=region=testLogSplittingAfterMasterRecoveryDueToZKExpiry,2,1365336889784.55884b67b9203a71bdf8c0e3de70452d., state=RS_ZK_REGION_OPENED, servername=asf001.sp2.ygridcore.net,43802,1365336889418, createTime=1365336890696, payload.length=0 2013-04-07 12:14:50,731 DEBUG [MASTER_OPEN_REGION-asf001.sp2.ygridcore.net,33796,1365336878036-0] zookeeper.ZKUtil(1680): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Retrieved 152 byte(s) of data from znode /hbase/region-in-transition/55884b67b9203a71bdf8c0e3de70452d; data=region=testLogSplittingAfterMasterRecoveryDueToZKExpiry,2,1365336889784.55884b67b9203a71bdf8c0e3de70452d., state=RS_ZK_REGION_OPENED, servername=asf001.sp2.ygridcore.net,43802,1365336889418, createTime=1365336890696, payload.length=0 2013-04-07 12:14:50,732 DEBUG [MASTER_OPEN_REGION-asf001.sp2.ygridcore.net,33796,1365336878036-1] zookeeper.ZKUtil(1680): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Retrieved 152 byte(s) of data from znode /hbase/region-in-transition/4e9a3d63237d29e6bf55bf2360834cbe; data=region=testLogSplittingAfterMasterRecoveryDueToZKExpiry,3,1365336889784.4e9a3d63237d29e6bf55bf2360834cbe., state=RS_ZK_REGION_OPENED, servername=asf001.sp2.ygridcore.net,60884,1365336878389, createTime=1365336890662, payload.length=0 2013-04-07 12:14:50,732 DEBUG [Thread-200-EventThread] zookeeper.ZooKeeperWatcher(274): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/region-in-transition/43207168ac66dce85cf2a4f916007450 2013-04-07 12:14:50,733 DEBUG [hbase-am-zkevent-worker-pool-20-thread-8] master.AssignmentManager$4(1159): The znode of region testLogSplittingAfterMasterRecoveryDueToZKExpiry,1,1365336889784.43207168ac66dce85cf2a4f916007450. has been deleted. 2013-04-07 12:14:50,733 DEBUG [hbase-am-zkevent-worker-pool-2-thread-5] master.AssignmentManager$4(1159): The znode of region testLogSplittingAfterMasterRecoveryDueToZKExpiry,1,1365336889784.43207168ac66dce85cf2a4f916007450. has been deleted. 2013-04-07 12:14:50,733 INFO [hbase-am-zkevent-worker-pool-20-thread-8] master.RegionStates(264): Region {NAME => 'testLogSplittingAfterMasterRecoveryDueToZKExpiry,1,1365336889784.43207168ac66dce85cf2a4f916007450.', STARTKEY => '1', ENDKEY => '2', ENCODED => 43207168ac66dce85cf2a4f916007450,} transitioned from {testLogSplittingAfterMasterRecoveryDueToZKExpiry,1,1365336889784.43207168ac66dce85cf2a4f916007450. state=OPEN, ts=1365336890704, server=asf001.sp2.ygridcore.net,43802,1365336889418} to {testLogSplittingAfterMasterRecoveryDueToZKExpiry,1,1365336889784.43207168ac66dce85cf2a4f916007450. state=OPEN, ts=1365336890733, server=asf001.sp2.ygridcore.net,43802,1365336889418} 2013-04-07 12:14:50,733 INFO [hbase-am-zkevent-worker-pool-2-thread-5] master.RegionStates(264): Region {NAME => 'testLogSplittingAfterMasterRecoveryDueToZKExpiry,1,1365336889784.43207168ac66dce85cf2a4f916007450.', STARTKEY => '1', ENDKEY => '2', ENCODED => 43207168ac66dce85cf2a4f916007450,} transitioned from {testLogSplittingAfterMasterRecoveryDueToZKExpiry,1,1365336889784.43207168ac66dce85cf2a4f916007450. state=OPEN, ts=1365336890703, server=asf001.sp2.ygridcore.net,43802,1365336889418} to {testLogSplittingAfterMasterRecoveryDueToZKExpiry,1,1365336889784.43207168ac66dce85cf2a4f916007450. state=OPEN, ts=1365336890733, server=asf001.sp2.ygridcore.net,43802,1365336889418} 2013-04-07 12:14:50,733 INFO [hbase-am-zkevent-worker-pool-20-thread-8] master.RegionStates(299): Onlined region {NAME => 'testLogSplittingAfterMasterRecoveryDueToZKExpiry,1,1365336889784.43207168ac66dce85cf2a4f916007450.', STARTKEY => '1', ENDKEY => '2', ENCODED => 43207168ac66dce85cf2a4f916007450,} on asf001.sp2.ygridcore.net,43802,1365336889418 2013-04-07 12:14:50,733 INFO [hbase-am-zkevent-worker-pool-2-thread-5] master.RegionStates(299): Onlined region {NAME => 'testLogSplittingAfterMasterRecoveryDueToZKExpiry,1,1365336889784.43207168ac66dce85cf2a4f916007450.', STARTKEY => '1', ENDKEY => '2', ENCODED => 43207168ac66dce85cf2a4f916007450,} on asf001.sp2.ygridcore.net,43802,1365336889418 2013-04-07 12:14:50,733 INFO [hbase-am-zkevent-worker-pool-20-thread-8] master.AssignmentManager$4(1164): The master has opened the region testLogSplittingAfterMasterRecoveryDueToZKExpiry,1,1365336889784.43207168ac66dce85cf2a4f916007450. that was online on asf001.sp2.ygridcore.net,43802,1365336889418 2013-04-07 12:14:50,734 INFO [hbase-am-zkevent-worker-pool-2-thread-5] master.AssignmentManager$4(1164): The master has opened the region testLogSplittingAfterMasterRecoveryDueToZKExpiry,1,1365336889784.43207168ac66dce85cf2a4f916007450. that was online on asf001.sp2.ygridcore.net,43802,1365336889418 2013-04-07 12:14:50,734 WARN [MASTER_OPEN_REGION-asf001.sp2.ygridcore.net,33796,1365336878036-3] zookeeper.RecoverableZooKeeper(161): Node /hbase/region-in-transition/43207168ac66dce85cf2a4f916007450 already deleted, retry=false 2013-04-07 12:14:50,734 WARN [MASTER_OPEN_REGION-asf001.sp2.ygridcore.net,33796,1365336878036-3] handler.OpenedRegionHandler(132): The znode of the region testLogSplittingAfterMasterRecoveryDueToZKExpiry,1,1365336889784.43207168ac66dce85cf2a4f916007450. would have already been deleted 2013-04-07 12:14:50,733 DEBUG [MASTER_OPEN_REGION-asf001.sp2.ygridcore.net,33796,1365336878036-2] zookeeper.ZKAssign(434): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Successfully deleted unassigned node for region 43207168ac66dce85cf2a4f916007450 in expected state RS_ZK_REGION_OPENED 2013-04-07 12:14:50,734 ERROR [MASTER_OPEN_REGION-asf001.sp2.ygridcore.net,33796,1365336878036-3] handler.OpenedRegionHandler(104): The znode of region testLogSplittingAfterMasterRecoveryDueToZKExpiry,1,1365336889784.43207168ac66dce85cf2a4f916007450. could not be deleted. 2013-04-07 12:14:50,734 DEBUG [MASTER_OPEN_REGION-asf001.sp2.ygridcore.net,33796,1365336878036-2] handler.OpenedRegionHandler(146): Handling OPENED event for testLogSplittingAfterMasterRecoveryDueToZKExpiry,3,1365336889784.4e9a3d63237d29e6bf55bf2360834cbe. from asf001.sp2.ygridcore.net,60884,1365336878389; deleting unassigned node 2013-04-07 12:14:50,734 DEBUG [MASTER_OPEN_REGION-asf001.sp2.ygridcore.net,33796,1365336878036-3] handler.OpenedRegionHandler(146): Handling OPENED event for testLogSplittingAfterMasterRecoveryDueToZKExpiry,5,1365336889784.5403a557a6168e5f3c5141eb494f6088. from asf001.sp2.ygridcore.net,60884,1365336878389; deleting unassigned node 2013-04-07 12:14:50,734 DEBUG [MASTER_OPEN_REGION-asf001.sp2.ygridcore.net,33796,1365336878036-2] zookeeper.ZKAssign(405): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Deleting existing unassigned node for 4e9a3d63237d29e6bf55bf2360834cbe that is in expected state RS_ZK_REGION_OPENED 2013-04-07 12:14:50,734 DEBUG [MASTER_OPEN_REGION-asf001.sp2.ygridcore.net,33796,1365336878036-3] zookeeper.ZKAssign(405): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Deleting existing unassigned node for 5403a557a6168e5f3c5141eb494f6088 that is in expected state RS_ZK_REGION_OPENED 2013-04-07 12:14:50,733 DEBUG [Thread-200-EventThread] zookeeper.ZooKeeperWatcher(274): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/region-in-transition 2013-04-07 12:14:50,735 DEBUG [hbase-am-zkevent-worker-pool-20-thread-7] zookeeper.ZKUtil(1680): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Retrieved 151 byte(s) of data from znode /hbase/region-in-transition/fb4182aef4ce07f011871ae0a083aee0 and set watcher; region=testLogSplittingAfterMasterRecoveryDueToZKExpiry,,1365336889784.fb4182aef4ce07f011871ae0a083aee0., state=RS_ZK_REGION_OPENED, servername=asf001.sp2.ygridcore.net,60884,1365336878389, createTime=1365336890710, payload.length=0 2013-04-07 12:14:50,735 DEBUG [hbase-am-zkevent-worker-pool-20-thread-7] master.AssignmentManager(740): Handling transition=RS_ZK_REGION_OPENED, server=asf001.sp2.ygridcore.net,60884,1365336878389, region=fb4182aef4ce07f011871ae0a083aee0, current state from region state map ={testLogSplittingAfterMasterRecoveryDueToZKExpiry,,1365336889784.fb4182aef4ce07f011871ae0a083aee0. state=OPENING, ts=1365336890719, server=asf001.sp2.ygridcore.net,60884,1365336878389} 2013-04-07 12:14:50,735 DEBUG [hbase-am-zkevent-worker-pool-2-thread-20] zookeeper.ZKUtil(1680): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Retrieved 151 byte(s) of data from znode /hbase/region-in-transition/fb4182aef4ce07f011871ae0a083aee0 and set watcher; region=testLogSplittingAfterMasterRecoveryDueToZKExpiry,,1365336889784.fb4182aef4ce07f011871ae0a083aee0., state=RS_ZK_REGION_OPENED, servername=asf001.sp2.ygridcore.net,60884,1365336878389, createTime=1365336890710, payload.length=0 2013-04-07 12:14:50,735 INFO [hbase-am-zkevent-worker-pool-20-thread-7] master.RegionStates(264): Region {NAME => 'testLogSplittingAfterMasterRecoveryDueToZKExpiry,,1365336889784.fb4182aef4ce07f011871ae0a083aee0.', STARTKEY => '', ENDKEY => '1', ENCODED => fb4182aef4ce07f011871ae0a083aee0,} transitioned from {testLogSplittingAfterMasterRecoveryDueToZKExpiry,,1365336889784.fb4182aef4ce07f011871ae0a083aee0. state=OPENING, ts=1365336890719, server=asf001.sp2.ygridcore.net,60884,1365336878389} to {testLogSplittingAfterMasterRecoveryDueToZKExpiry,,1365336889784.fb4182aef4ce07f011871ae0a083aee0. state=OPEN, ts=1365336890735, server=asf001.sp2.ygridcore.net,60884,1365336878389} 2013-04-07 12:14:50,735 DEBUG [hbase-am-zkevent-worker-pool-2-thread-20] master.AssignmentManager(740): Handling transition=RS_ZK_REGION_OPENED, server=asf001.sp2.ygridcore.net,60884,1365336878389, region=fb4182aef4ce07f011871ae0a083aee0, current state from region state map ={testLogSplittingAfterMasterRecoveryDueToZKExpiry,,1365336889784.fb4182aef4ce07f011871ae0a083aee0. state=OPEN, ts=1365336890727, server=asf001.sp2.ygridcore.net,60884,1365336878389} 2013-04-07 12:14:50,736 WARN [hbase-am-zkevent-worker-pool-2-thread-20] master.AssignmentManager(934): Received OPENED for region fb4182aef4ce07f011871ae0a083aee0 from server asf001.sp2.ygridcore.net,60884,1365336878389 but region was in the state {testLogSplittingAfterMasterRecoveryDueToZKExpiry,,1365336889784.fb4182aef4ce07f011871ae0a083aee0. state=OPEN, ts=1365336890727, server=asf001.sp2.ygridcore.net,60884,1365336878389} and not in expected PENDING_OPEN or OPENING states, or not on the expected server 2013-04-07 12:14:50,736 DEBUG [Thread-200-EventThread] zookeeper.ZooKeeperWatcher(274): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/region-in-transition/55884b67b9203a71bdf8c0e3de70452d 2013-04-07 12:14:50,736 DEBUG [hbase-am-zkevent-worker-pool-20-thread-10] master.AssignmentManager$4(1159): The znode of region testLogSplittingAfterMasterRecoveryDueToZKExpiry,2,1365336889784.55884b67b9203a71bdf8c0e3de70452d. has been deleted. 2013-04-07 12:14:50,736 DEBUG [MASTER_OPEN_REGION-asf001.sp2.ygridcore.net,33796,1365336878036-4] zookeeper.ZKAssign(434): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Successfully deleted unassigned node for region 55884b67b9203a71bdf8c0e3de70452d in expected state RS_ZK_REGION_OPENED 2013-04-07 12:14:50,736 INFO [hbase-am-zkevent-worker-pool-20-thread-10] master.RegionStates(264): Region {NAME => 'testLogSplittingAfterMasterRecoveryDueToZKExpiry,2,1365336889784.55884b67b9203a71bdf8c0e3de70452d.', STARTKEY => '2', ENDKEY => '3', ENCODED => 55884b67b9203a71bdf8c0e3de70452d,} transitioned from {testLogSplittingAfterMasterRecoveryDueToZKExpiry,2,1365336889784.55884b67b9203a71bdf8c0e3de70452d. state=OPEN, ts=1365336890708, server=asf001.sp2.ygridcore.net,43802,1365336889418} to {testLogSplittingAfterMasterRecoveryDueToZKExpiry,2,1365336889784.55884b67b9203a71bdf8c0e3de70452d. state=OPEN, ts=1365336890736, server=asf001.sp2.ygridcore.net,43802,1365336889418} 2013-04-07 12:14:50,736 INFO [hbase-am-zkevent-worker-pool-20-thread-10] master.RegionStates(299): Onlined region {NAME => 'testLogSplittingAfterMasterRecoveryDueToZKExpiry,2,1365336889784.55884b67b9203a71bdf8c0e3de70452d.', STARTKEY => '2', ENDKEY => '3', ENCODED => 55884b67b9203a71bdf8c0e3de70452d,} on asf001.sp2.ygridcore.net,43802,1365336889418 2013-04-07 12:14:50,736 DEBUG [hbase-am-zkevent-worker-pool-2-thread-4] master.AssignmentManager$4(1159): The znode of region testLogSplittingAfterMasterRecoveryDueToZKExpiry,2,1365336889784.55884b67b9203a71bdf8c0e3de70452d. has been deleted. 2013-04-07 12:14:50,737 INFO [hbase-am-zkevent-worker-pool-20-thread-10] master.AssignmentManager$4(1164): The master has opened the region testLogSplittingAfterMasterRecoveryDueToZKExpiry,2,1365336889784.55884b67b9203a71bdf8c0e3de70452d. that was online on asf001.sp2.ygridcore.net,43802,1365336889418 2013-04-07 12:14:50,737 INFO [hbase-am-zkevent-worker-pool-2-thread-4] master.RegionStates(264): Region {NAME => 'testLogSplittingAfterMasterRecoveryDueToZKExpiry,2,1365336889784.55884b67b9203a71bdf8c0e3de70452d.', STARTKEY => '2', ENDKEY => '3', ENCODED => 55884b67b9203a71bdf8c0e3de70452d,} transitioned from {testLogSplittingAfterMasterRecoveryDueToZKExpiry,2,1365336889784.55884b67b9203a71bdf8c0e3de70452d. state=OPEN, ts=1365336890709, server=asf001.sp2.ygridcore.net,43802,1365336889418} to {testLogSplittingAfterMasterRecoveryDueToZKExpiry,2,1365336889784.55884b67b9203a71bdf8c0e3de70452d. state=OPEN, ts=1365336890737, server=asf001.sp2.ygridcore.net,43802,1365336889418} 2013-04-07 12:14:50,737 INFO [hbase-am-zkevent-worker-pool-2-thread-4] master.RegionStates(299): Onlined region {NAME => 'testLogSplittingAfterMasterRecoveryDueToZKExpiry,2,1365336889784.55884b67b9203a71bdf8c0e3de70452d.', STARTKEY => '2', ENDKEY => '3', ENCODED => 55884b67b9203a71bdf8c0e3de70452d,} on asf001.sp2.ygridcore.net,43802,1365336889418 2013-04-07 12:14:50,737 INFO [hbase-am-zkevent-worker-pool-2-thread-4] master.AssignmentManager$4(1164): The master has opened the region testLogSplittingAfterMasterRecoveryDueToZKExpiry,2,1365336889784.55884b67b9203a71bdf8c0e3de70452d. that was online on asf001.sp2.ygridcore.net,43802,1365336889418 2013-04-07 12:14:50,737 WARN [MASTER_OPEN_REGION-asf001.sp2.ygridcore.net,33796,1365336878036-0] zookeeper.RecoverableZooKeeper(161): Node /hbase/region-in-transition/55884b67b9203a71bdf8c0e3de70452d already deleted, retry=false 2013-04-07 12:14:50,737 DEBUG [MASTER_OPEN_REGION-asf001.sp2.ygridcore.net,33796,1365336878036-4] handler.OpenedRegionHandler(146): Handling OPENED event for testLogSplittingAfterMasterRecoveryDueToZKExpiry,5,1365336889784.5403a557a6168e5f3c5141eb494f6088. from asf001.sp2.ygridcore.net,60884,1365336878389; deleting unassigned node 2013-04-07 12:14:50,737 WARN [MASTER_OPEN_REGION-asf001.sp2.ygridcore.net,33796,1365336878036-0] handler.OpenedRegionHandler(132): The znode of the region testLogSplittingAfterMasterRecoveryDueToZKExpiry,2,1365336889784.55884b67b9203a71bdf8c0e3de70452d. would have already been deleted 2013-04-07 12:14:50,737 DEBUG [Thread-200-EventThread] zookeeper.ZooKeeperWatcher(274): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/region-in-transition/4e9a3d63237d29e6bf55bf2360834cbe 2013-04-07 12:14:50,738 INFO [PRI IPC Server handler 4 on 60884] regionserver.HRegionServer(3434): Received close region: fb4182aef4ce07f011871ae0a083aee0Transitioning in ZK: no. Version of ZK closing node:-1. Destination server:null 2013-04-07 12:14:50,737 ERROR [MASTER_OPEN_REGION-asf001.sp2.ygridcore.net,33796,1365336878036-0] handler.OpenedRegionHandler(104): The znode of region testLogSplittingAfterMasterRecoveryDueToZKExpiry,2,1365336889784.55884b67b9203a71bdf8c0e3de70452d. could not be deleted. 2013-04-07 12:14:50,738 DEBUG [MASTER_OPEN_REGION-asf001.sp2.ygridcore.net,33796,1365336878036-1] zookeeper.ZKAssign(434): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Successfully deleted unassigned node for region 4e9a3d63237d29e6bf55bf2360834cbe in expected state RS_ZK_REGION_OPENED 2013-04-07 12:14:50,738 DEBUG [hbase-am-zkevent-worker-pool-2-thread-6] master.AssignmentManager$4(1159): The znode of region testLogSplittingAfterMasterRecoveryDueToZKExpiry,3,1365336889784.4e9a3d63237d29e6bf55bf2360834cbe. has been deleted. 2013-04-07 12:14:50,738 DEBUG [hbase-am-zkevent-worker-pool-20-thread-11] master.AssignmentManager$4(1159): The znode of region testLogSplittingAfterMasterRecoveryDueToZKExpiry,3,1365336889784.4e9a3d63237d29e6bf55bf2360834cbe. has been deleted. 2013-04-07 12:14:50,738 INFO [hbase-am-zkevent-worker-pool-2-thread-6] master.RegionStates(264): Region {NAME => 'testLogSplittingAfterMasterRecoveryDueToZKExpiry,3,1365336889784.4e9a3d63237d29e6bf55bf2360834cbe.', STARTKEY => '3', ENDKEY => '4', ENCODED => 4e9a3d63237d29e6bf55bf2360834cbe,} transitioned from {testLogSplittingAfterMasterRecoveryDueToZKExpiry,3,1365336889784.4e9a3d63237d29e6bf55bf2360834cbe. state=OPEN, ts=1365336890712, server=asf001.sp2.ygridcore.net,60884,1365336878389} to {testLogSplittingAfterMasterRecoveryDueToZKExpiry,3,1365336889784.4e9a3d63237d29e6bf55bf2360834cbe. state=OPEN, ts=1365336890738, server=asf001.sp2.ygridcore.net,60884,1365336878389} 2013-04-07 12:14:50,738 INFO [hbase-am-zkevent-worker-pool-20-thread-11] master.RegionStates(264): Region {NAME => 'testLogSplittingAfterMasterRecoveryDueToZKExpiry,3,1365336889784.4e9a3d63237d29e6bf55bf2360834cbe.', STARTKEY => '3', ENDKEY => '4', ENCODED => 4e9a3d63237d29e6bf55bf2360834cbe,} transitioned from {testLogSplittingAfterMasterRecoveryDueToZKExpiry,3,1365336889784.4e9a3d63237d29e6bf55bf2360834cbe. state=OPEN, ts=1365336890711, server=asf001.sp2.ygridcore.net,60884,1365336878389} to {testLogSplittingAfterMasterRecoveryDueToZKExpiry,3,1365336889784.4e9a3d63237d29e6bf55bf2360834cbe. state=OPEN, ts=1365336890738, server=asf001.sp2.ygridcore.net,60884,1365336878389} 2013-04-07 12:14:50,738 INFO [hbase-am-zkevent-worker-pool-2-thread-6] master.RegionStates(299): Onlined region {NAME => 'testLogSplittingAfterMasterRecoveryDueToZKExpiry,3,1365336889784.4e9a3d63237d29e6bf55bf2360834cbe.', STARTKEY => '3', ENDKEY => '4', ENCODED => 4e9a3d63237d29e6bf55bf2360834cbe,} on asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:14:50,738 INFO [hbase-am-zkevent-worker-pool-20-thread-11] master.RegionStates(299): Onlined region {NAME => 'testLogSplittingAfterMasterRecoveryDueToZKExpiry,3,1365336889784.4e9a3d63237d29e6bf55bf2360834cbe.', STARTKEY => '3', ENDKEY => '4', ENCODED => 4e9a3d63237d29e6bf55bf2360834cbe,} on asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:14:50,737 DEBUG [MASTER_OPEN_REGION-asf001.sp2.ygridcore.net,33796,1365336878036-4] zookeeper.ZKAssign(405): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Deleting existing unassigned node for 5403a557a6168e5f3c5141eb494f6088 that is in expected state RS_ZK_REGION_OPENED 2013-04-07 12:14:50,739 INFO [hbase-am-zkevent-worker-pool-20-thread-11] master.AssignmentManager$4(1164): The master has opened the region testLogSplittingAfterMasterRecoveryDueToZKExpiry,3,1365336889784.4e9a3d63237d29e6bf55bf2360834cbe. that was online on asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:14:50,739 INFO [hbase-am-zkevent-worker-pool-2-thread-6] master.AssignmentManager$4(1164): The master has opened the region testLogSplittingAfterMasterRecoveryDueToZKExpiry,3,1365336889784.4e9a3d63237d29e6bf55bf2360834cbe. that was online on asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:14:50,739 DEBUG [RS_CLOSE_REGION-asf001.sp2.ygridcore.net,60884,1365336878389-0] handler.CloseRegionHandler(125): Processing close of testLogSplittingAfterMasterRecoveryDueToZKExpiry,,1365336889784.fb4182aef4ce07f011871ae0a083aee0. 2013-04-07 12:14:50,739 DEBUG [MASTER_OPEN_REGION-asf001.sp2.ygridcore.net,33796,1365336878036-2] zookeeper.ZKUtil(775): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Unable to get data of znode /hbase/region-in-transition/4e9a3d63237d29e6bf55bf2360834cbe because node does not exist (not necessarily an error) 2013-04-07 12:14:50,738 DEBUG [MASTER_OPEN_REGION-asf001.sp2.ygridcore.net,33796,1365336878036-1] handler.OpenedRegionHandler(146): Handling OPENED event for testLogSplittingAfterMasterRecoveryDueToZKExpiry,,1365336889784.fb4182aef4ce07f011871ae0a083aee0. from asf001.sp2.ygridcore.net,60884,1365336878389; deleting unassigned node 2013-04-07 12:14:50,738 DEBUG [MASTER_OPEN_REGION-asf001.sp2.ygridcore.net,33796,1365336878036-0] handler.OpenedRegionHandler(146): Handling OPENED event for testLogSplittingAfterMasterRecoveryDueToZKExpiry,,1365336889784.fb4182aef4ce07f011871ae0a083aee0. from asf001.sp2.ygridcore.net,60884,1365336878389; deleting unassigned node 2013-04-07 12:14:50,740 DEBUG [MASTER_OPEN_REGION-asf001.sp2.ygridcore.net,33796,1365336878036-1] zookeeper.ZKAssign(405): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Deleting existing unassigned node for fb4182aef4ce07f011871ae0a083aee0 that is in expected state RS_ZK_REGION_OPENED 2013-04-07 12:14:50,740 WARN [MASTER_OPEN_REGION-asf001.sp2.ygridcore.net,33796,1365336878036-2] handler.OpenedRegionHandler(132): The znode of the region testLogSplittingAfterMasterRecoveryDueToZKExpiry,3,1365336889784.4e9a3d63237d29e6bf55bf2360834cbe. would have already been deleted 2013-04-07 12:14:50,740 DEBUG [RS_CLOSE_REGION-asf001.sp2.ygridcore.net,60884,1365336878389-0] regionserver.HRegion(889): Closing testLogSplittingAfterMasterRecoveryDueToZKExpiry,,1365336889784.fb4182aef4ce07f011871ae0a083aee0.: disabling compactions & flushes 2013-04-07 12:14:50,740 ERROR [MASTER_OPEN_REGION-asf001.sp2.ygridcore.net,33796,1365336878036-2] handler.OpenedRegionHandler(104): The znode of region testLogSplittingAfterMasterRecoveryDueToZKExpiry,3,1365336889784.4e9a3d63237d29e6bf55bf2360834cbe. could not be deleted. 2013-04-07 12:14:50,740 DEBUG [MASTER_OPEN_REGION-asf001.sp2.ygridcore.net,33796,1365336878036-3] zookeeper.ZKUtil(1680): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Retrieved 152 byte(s) of data from znode /hbase/region-in-transition/5403a557a6168e5f3c5141eb494f6088; data=region=testLogSplittingAfterMasterRecoveryDueToZKExpiry,5,1365336889784.5403a557a6168e5f3c5141eb494f6088., state=RS_ZK_REGION_OPENED, servername=asf001.sp2.ygridcore.net,60884,1365336878389, createTime=1365336890687, payload.length=0 2013-04-07 12:14:50,740 DEBUG [MASTER_OPEN_REGION-asf001.sp2.ygridcore.net,33796,1365336878036-0] zookeeper.ZKAssign(405): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Deleting existing unassigned node for fb4182aef4ce07f011871ae0a083aee0 that is in expected state RS_ZK_REGION_OPENED 2013-04-07 12:14:50,741 DEBUG [RS_CLOSE_REGION-asf001.sp2.ygridcore.net,60884,1365336878389-0] regionserver.HRegion(911): Updates disabled for region testLogSplittingAfterMasterRecoveryDueToZKExpiry,,1365336889784.fb4182aef4ce07f011871ae0a083aee0. 2013-04-07 12:14:50,741 DEBUG [hbase-am-zkevent-worker-pool-2-thread-20] master.AssignmentManager(1638): Sent CLOSE to asf001.sp2.ygridcore.net,60884,1365336878389 for region testLogSplittingAfterMasterRecoveryDueToZKExpiry,,1365336889784.fb4182aef4ce07f011871ae0a083aee0. 2013-04-07 12:14:50,742 INFO [StoreCloserThread-testLogSplittingAfterMasterRecoveryDueToZKExpiry,,1365336889784.fb4182aef4ce07f011871ae0a083aee0.-1] regionserver.HStore(622): Closed col 2013-04-07 12:14:50,743 INFO [RS_CLOSE_REGION-asf001.sp2.ygridcore.net,60884,1365336878389-0] regionserver.HRegion(968): Closed testLogSplittingAfterMasterRecoveryDueToZKExpiry,,1365336889784.fb4182aef4ce07f011871ae0a083aee0. 2013-04-07 12:14:50,743 DEBUG [RS_CLOSE_REGION-asf001.sp2.ygridcore.net,60884,1365336878389-0] handler.CloseRegionHandler(177): Closed region testLogSplittingAfterMasterRecoveryDueToZKExpiry,,1365336889784.fb4182aef4ce07f011871ae0a083aee0. 2013-04-07 12:14:50,744 DEBUG [MASTER_OPEN_REGION-asf001.sp2.ygridcore.net,33796,1365336878036-4] zookeeper.ZKUtil(1680): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Retrieved 152 byte(s) of data from znode /hbase/region-in-transition/5403a557a6168e5f3c5141eb494f6088; data=region=testLogSplittingAfterMasterRecoveryDueToZKExpiry,5,1365336889784.5403a557a6168e5f3c5141eb494f6088., state=RS_ZK_REGION_OPENED, servername=asf001.sp2.ygridcore.net,60884,1365336878389, createTime=1365336890687, payload.length=0 2013-04-07 12:14:50,744 DEBUG [MASTER_OPEN_REGION-asf001.sp2.ygridcore.net,33796,1365336878036-1] zookeeper.ZKUtil(1680): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Retrieved 151 byte(s) of data from znode /hbase/region-in-transition/fb4182aef4ce07f011871ae0a083aee0; data=region=testLogSplittingAfterMasterRecoveryDueToZKExpiry,,1365336889784.fb4182aef4ce07f011871ae0a083aee0., state=RS_ZK_REGION_OPENED, servername=asf001.sp2.ygridcore.net,60884,1365336878389, createTime=1365336890710, payload.length=0 2013-04-07 12:14:50,745 DEBUG [MASTER_OPEN_REGION-asf001.sp2.ygridcore.net,33796,1365336878036-0] zookeeper.ZKUtil(1680): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Retrieved 151 byte(s) of data from znode /hbase/region-in-transition/fb4182aef4ce07f011871ae0a083aee0; data=region=testLogSplittingAfterMasterRecoveryDueToZKExpiry,,1365336889784.fb4182aef4ce07f011871ae0a083aee0., state=RS_ZK_REGION_OPENED, servername=asf001.sp2.ygridcore.net,60884,1365336878389, createTime=1365336890710, payload.length=0 2013-04-07 12:14:50,745 DEBUG [Thread-200-EventThread] zookeeper.ZooKeeperWatcher(274): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/region-in-transition/5403a557a6168e5f3c5141eb494f6088 2013-04-07 12:14:50,746 DEBUG [hbase-am-zkevent-worker-pool-20-thread-12] master.AssignmentManager$4(1159): The znode of region testLogSplittingAfterMasterRecoveryDueToZKExpiry,5,1365336889784.5403a557a6168e5f3c5141eb494f6088. has been deleted. 2013-04-07 12:14:50,746 DEBUG [hbase-am-zkevent-worker-pool-2-thread-8] master.AssignmentManager$4(1159): The znode of region testLogSplittingAfterMasterRecoveryDueToZKExpiry,5,1365336889784.5403a557a6168e5f3c5141eb494f6088. has been deleted. 2013-04-07 12:14:50,746 INFO [hbase-am-zkevent-worker-pool-20-thread-12] master.RegionStates(264): Region {NAME => 'testLogSplittingAfterMasterRecoveryDueToZKExpiry,5,1365336889784.5403a557a6168e5f3c5141eb494f6088.', STARTKEY => '5', ENDKEY => '', ENCODED => 5403a557a6168e5f3c5141eb494f6088,} transitioned from {testLogSplittingAfterMasterRecoveryDueToZKExpiry,5,1365336889784.5403a557a6168e5f3c5141eb494f6088. state=OPEN, ts=1365336890719, server=asf001.sp2.ygridcore.net,60884,1365336878389} to {testLogSplittingAfterMasterRecoveryDueToZKExpiry,5,1365336889784.5403a557a6168e5f3c5141eb494f6088. state=OPEN, ts=1365336890746, server=asf001.sp2.ygridcore.net,60884,1365336878389} 2013-04-07 12:14:50,746 INFO [hbase-am-zkevent-worker-pool-2-thread-8] master.RegionStates(264): Region {NAME => 'testLogSplittingAfterMasterRecoveryDueToZKExpiry,5,1365336889784.5403a557a6168e5f3c5141eb494f6088.', STARTKEY => '5', ENDKEY => '', ENCODED => 5403a557a6168e5f3c5141eb494f6088,} transitioned from {testLogSplittingAfterMasterRecoveryDueToZKExpiry,5,1365336889784.5403a557a6168e5f3c5141eb494f6088. state=OPEN, ts=1365336890718, server=asf001.sp2.ygridcore.net,60884,1365336878389} to {testLogSplittingAfterMasterRecoveryDueToZKExpiry,5,1365336889784.5403a557a6168e5f3c5141eb494f6088. state=OPEN, ts=1365336890746, server=asf001.sp2.ygridcore.net,60884,1365336878389} 2013-04-07 12:14:50,746 DEBUG [MASTER_OPEN_REGION-asf001.sp2.ygridcore.net,33796,1365336878036-3] zookeeper.ZKAssign(434): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Successfully deleted unassigned node for region 5403a557a6168e5f3c5141eb494f6088 in expected state RS_ZK_REGION_OPENED 2013-04-07 12:14:50,746 INFO [hbase-am-zkevent-worker-pool-2-thread-8] master.RegionStates(299): Onlined region {NAME => 'testLogSplittingAfterMasterRecoveryDueToZKExpiry,5,1365336889784.5403a557a6168e5f3c5141eb494f6088.', STARTKEY => '5', ENDKEY => '', ENCODED => 5403a557a6168e5f3c5141eb494f6088,} on asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:14:50,747 WARN [MASTER_OPEN_REGION-asf001.sp2.ygridcore.net,33796,1365336878036-4] zookeeper.RecoverableZooKeeper(161): Node /hbase/region-in-transition/5403a557a6168e5f3c5141eb494f6088 already deleted, retry=false 2013-04-07 12:14:50,747 INFO [hbase-am-zkevent-worker-pool-2-thread-8] master.AssignmentManager$4(1164): The master has opened the region testLogSplittingAfterMasterRecoveryDueToZKExpiry,5,1365336889784.5403a557a6168e5f3c5141eb494f6088. that was online on asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:14:50,747 WARN [MASTER_OPEN_REGION-asf001.sp2.ygridcore.net,33796,1365336878036-4] handler.OpenedRegionHandler(132): The znode of the region testLogSplittingAfterMasterRecoveryDueToZKExpiry,5,1365336889784.5403a557a6168e5f3c5141eb494f6088. would have already been deleted 2013-04-07 12:14:50,746 INFO [hbase-am-zkevent-worker-pool-20-thread-12] master.RegionStates(299): Onlined region {NAME => 'testLogSplittingAfterMasterRecoveryDueToZKExpiry,5,1365336889784.5403a557a6168e5f3c5141eb494f6088.', STARTKEY => '5', ENDKEY => '', ENCODED => 5403a557a6168e5f3c5141eb494f6088,} on asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:14:50,746 DEBUG [Thread-200-EventThread] zookeeper.ZooKeeperWatcher(274): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/region-in-transition 2013-04-07 12:14:50,747 INFO [hbase-am-zkevent-worker-pool-20-thread-12] master.AssignmentManager$4(1164): The master has opened the region testLogSplittingAfterMasterRecoveryDueToZKExpiry,5,1365336889784.5403a557a6168e5f3c5141eb494f6088. that was online on asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:14:50,747 DEBUG [MASTER_OPEN_REGION-asf001.sp2.ygridcore.net,33796,1365336878036-1] zookeeper.ZKAssign(434): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Successfully deleted unassigned node for region fb4182aef4ce07f011871ae0a083aee0 in expected state RS_ZK_REGION_OPENED 2013-04-07 12:14:50,747 DEBUG [Thread-200-EventThread] zookeeper.ZooKeeperWatcher(274): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/region-in-transition/fb4182aef4ce07f011871ae0a083aee0 2013-04-07 12:14:50,747 ERROR [MASTER_OPEN_REGION-asf001.sp2.ygridcore.net,33796,1365336878036-4] handler.OpenedRegionHandler(104): The znode of region testLogSplittingAfterMasterRecoveryDueToZKExpiry,5,1365336889784.5403a557a6168e5f3c5141eb494f6088. could not be deleted. 2013-04-07 12:14:50,748 DEBUG [hbase-am-zkevent-worker-pool-20-thread-14] master.AssignmentManager$4(1159): The znode of region testLogSplittingAfterMasterRecoveryDueToZKExpiry,,1365336889784.fb4182aef4ce07f011871ae0a083aee0. has been deleted. 2013-04-07 12:14:50,748 DEBUG [hbase-am-zkevent-worker-pool-2-thread-10] master.AssignmentManager$4(1159): The znode of region testLogSplittingAfterMasterRecoveryDueToZKExpiry,,1365336889784.fb4182aef4ce07f011871ae0a083aee0. has been deleted. 2013-04-07 12:14:50,748 INFO [hbase-am-zkevent-worker-pool-20-thread-14] master.RegionStates(264): Region {NAME => 'testLogSplittingAfterMasterRecoveryDueToZKExpiry,,1365336889784.fb4182aef4ce07f011871ae0a083aee0.', STARTKEY => '', ENDKEY => '1', ENCODED => fb4182aef4ce07f011871ae0a083aee0,} transitioned from {testLogSplittingAfterMasterRecoveryDueToZKExpiry,,1365336889784.fb4182aef4ce07f011871ae0a083aee0. state=OPEN, ts=1365336890735, server=asf001.sp2.ygridcore.net,60884,1365336878389} to {testLogSplittingAfterMasterRecoveryDueToZKExpiry,,1365336889784.fb4182aef4ce07f011871ae0a083aee0. state=OPEN, ts=1365336890748, server=asf001.sp2.ygridcore.net,60884,1365336878389} 2013-04-07 12:14:50,748 INFO [hbase-am-zkevent-worker-pool-2-thread-10] master.RegionStates(264): Region {NAME => 'testLogSplittingAfterMasterRecoveryDueToZKExpiry,,1365336889784.fb4182aef4ce07f011871ae0a083aee0.', STARTKEY => '', ENDKEY => '1', ENCODED => fb4182aef4ce07f011871ae0a083aee0,} transitioned from {testLogSplittingAfterMasterRecoveryDueToZKExpiry,,1365336889784.fb4182aef4ce07f011871ae0a083aee0. state=OPEN, ts=1365336890727, server=asf001.sp2.ygridcore.net,60884,1365336878389} to {testLogSplittingAfterMasterRecoveryDueToZKExpiry,,1365336889784.fb4182aef4ce07f011871ae0a083aee0. state=OPEN, ts=1365336890748, server=asf001.sp2.ygridcore.net,60884,1365336878389} 2013-04-07 12:14:50,748 INFO [hbase-am-zkevent-worker-pool-20-thread-14] master.RegionStates(299): Onlined region {NAME => 'testLogSplittingAfterMasterRecoveryDueToZKExpiry,,1365336889784.fb4182aef4ce07f011871ae0a083aee0.', STARTKEY => '', ENDKEY => '1', ENCODED => fb4182aef4ce07f011871ae0a083aee0,} on asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:14:50,748 INFO [hbase-am-zkevent-worker-pool-2-thread-10] master.RegionStates(299): Onlined region {NAME => 'testLogSplittingAfterMasterRecoveryDueToZKExpiry,,1365336889784.fb4182aef4ce07f011871ae0a083aee0.', STARTKEY => '', ENDKEY => '1', ENCODED => fb4182aef4ce07f011871ae0a083aee0,} on asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:14:50,749 INFO [hbase-am-zkevent-worker-pool-20-thread-14] master.AssignmentManager$4(1164): The master has opened the region testLogSplittingAfterMasterRecoveryDueToZKExpiry,,1365336889784.fb4182aef4ce07f011871ae0a083aee0. that was online on asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:14:50,749 INFO [hbase-am-zkevent-worker-pool-2-thread-10] master.AssignmentManager$4(1164): The master has opened the region testLogSplittingAfterMasterRecoveryDueToZKExpiry,,1365336889784.fb4182aef4ce07f011871ae0a083aee0. that was online on asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:14:50,749 WARN [MASTER_OPEN_REGION-asf001.sp2.ygridcore.net,33796,1365336878036-0] zookeeper.RecoverableZooKeeper(161): Node /hbase/region-in-transition/fb4182aef4ce07f011871ae0a083aee0 already deleted, retry=false 2013-04-07 12:14:50,749 WARN [MASTER_OPEN_REGION-asf001.sp2.ygridcore.net,33796,1365336878036-0] handler.OpenedRegionHandler(132): The znode of the region testLogSplittingAfterMasterRecoveryDueToZKExpiry,,1365336889784.fb4182aef4ce07f011871ae0a083aee0. would have already been deleted 2013-04-07 12:14:50,749 ERROR [MASTER_OPEN_REGION-asf001.sp2.ygridcore.net,33796,1365336878036-0] handler.OpenedRegionHandler(104): The znode of region testLogSplittingAfterMasterRecoveryDueToZKExpiry,,1365336889784.fb4182aef4ce07f011871ae0a083aee0. could not be deleted. 2013-04-07 12:16:02,353 DEBUG [Thread-200-EventThread] zookeeper.ZooKeeperWatcher(274): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2013-04-07 12:16:02,354 DEBUG [Thread-200-EventThread] zookeeper.ZKUtil(422): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Set watcher on existing znode /hbase/rs/asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:16:02,355 DEBUG [Thread-200-EventThread] zookeeper.ZKUtil(422): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Set watcher on existing znode /hbase/rs/asf001.sp2.ygridcore.net,43802,1365336889418 2013-04-07 12:16:02,356 DEBUG [Thread-200-EventThread] zookeeper.ZKUtil(422): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Set watcher on existing znode /hbase/rs/asf001.sp2.ygridcore.net,49518,1365336962323 2013-04-07 12:16:02,358 DEBUG [Thread-200-EventThread] zookeeper.ZKUtil(422): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Set watcher on existing znode /hbase/rs/asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:16:02,359 DEBUG [Thread-200-EventThread] zookeeper.ZKUtil(422): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Set watcher on existing znode /hbase/rs/asf001.sp2.ygridcore.net,43802,1365336889418 2013-04-07 12:16:02,359 DEBUG [Thread-200-EventThread] zookeeper.ZKUtil(422): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Set watcher on existing znode /hbase/rs/asf001.sp2.ygridcore.net,49518,1365336962323 2013-04-07 12:16:02,994 DEBUG [Thread-200-EventThread] zookeeper.ZooKeeperWatcher(274): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/region-in-transition 2013-04-07 12:16:02,994 INFO [asf001.sp2.ygridcore.net,33796,1365336878036-GeneralBulkAssigner-1] master.RegionStates(264): Region {NAME => 'testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,g,1365336962398.682475e1358b7306bd3406e9330263a4.', STARTKEY => 'g', ENDKEY => 'h', ENCODED => 682475e1358b7306bd3406e9330263a4,} transitioned from {testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,g,1365336962398.682475e1358b7306bd3406e9330263a4. state=OFFLINE, ts=1365336962989, server=null} to {testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,g,1365336962398.682475e1358b7306bd3406e9330263a4. state=OFFLINE, ts=1365336962994, server=null} 2013-04-07 12:16:02,994 DEBUG [asf001.sp2.ygridcore.net,33796,1365336878036-GeneralBulkAssigner-0] zookeeper.ZKAssign(177): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Async create of unassigned node for 1ce6d62427641e2a0e135d3d27b54b5d with OFFLINE state 2013-04-07 12:16:02,995 DEBUG [asf001.sp2.ygridcore.net,33796,1365336878036-GeneralBulkAssigner-1] zookeeper.ZKAssign(177): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Async create of unassigned node for 682475e1358b7306bd3406e9330263a4 with OFFLINE state 2013-04-07 12:16:02,995 INFO [asf001.sp2.ygridcore.net,33796,1365336878036-GeneralBulkAssigner-2] master.RegionStates(264): Region {NAME => 'testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,i,1365336962398.385e5846630b0cb4493758961a2c2457.', STARTKEY => 'i', ENDKEY => 'j', ENCODED => 385e5846630b0cb4493758961a2c2457,} transitioned from {testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,i,1365336962398.385e5846630b0cb4493758961a2c2457. state=OFFLINE, ts=1365336962989, server=null} to {testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,i,1365336962398.385e5846630b0cb4493758961a2c2457. state=OFFLINE, ts=1365336962995, server=null} 2013-04-07 12:16:02,995 DEBUG [asf001.sp2.ygridcore.net,33796,1365336878036-GeneralBulkAssigner-2] zookeeper.ZKAssign(177): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Async create of unassigned node for 385e5846630b0cb4493758961a2c2457 with OFFLINE state 2013-04-07 12:16:02,995 INFO [asf001.sp2.ygridcore.net,33796,1365336878036-GeneralBulkAssigner-1] master.RegionStates(264): Region {NAME => 'testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,a,1365336962398.b1b340dff1e06fd6cb0b0a9c64964551.', STARTKEY => 'a', ENDKEY => 'b', ENCODED => b1b340dff1e06fd6cb0b0a9c64964551,} transitioned from {testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,a,1365336962398.b1b340dff1e06fd6cb0b0a9c64964551. state=OFFLINE, ts=1365336962989, server=null} to {testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,a,1365336962398.b1b340dff1e06fd6cb0b0a9c64964551. state=OFFLINE, ts=1365336962995, server=null} 2013-04-07 12:16:02,995 DEBUG [asf001.sp2.ygridcore.net,33796,1365336878036-GeneralBulkAssigner-1] zookeeper.ZKAssign(177): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Async create of unassigned node for b1b340dff1e06fd6cb0b0a9c64964551 with OFFLINE state 2013-04-07 12:16:02,995 INFO [asf001.sp2.ygridcore.net,33796,1365336878036-GeneralBulkAssigner-2] master.RegionStates(264): Region {NAME => 'testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,b,1365336962398.11c8123e8d563f264d947ffa106e4c7c.', STARTKEY => 'b', ENDKEY => 'c', ENCODED => 11c8123e8d563f264d947ffa106e4c7c,} transitioned from {testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,b,1365336962398.11c8123e8d563f264d947ffa106e4c7c. state=OFFLINE, ts=1365336962989, server=null} to {testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,b,1365336962398.11c8123e8d563f264d947ffa106e4c7c. state=OFFLINE, ts=1365336962995, server=null} 2013-04-07 12:16:02,995 DEBUG [Thread-200-EventThread] master.OfflineCallback(69): rs={testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,d,1365336962398.e99798d2c4d566ddb063efe03842457f. state=OFFLINE, ts=1365336962989, server=null}, server=asf001.sp2.ygridcore.net,49518,1365336962323 2013-04-07 12:16:02,995 DEBUG [asf001.sp2.ygridcore.net,33796,1365336878036-GeneralBulkAssigner-2] zookeeper.ZKAssign(177): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Async create of unassigned node for 11c8123e8d563f264d947ffa106e4c7c with OFFLINE state 2013-04-07 12:16:02,996 INFO [asf001.sp2.ygridcore.net,33796,1365336878036-GeneralBulkAssigner-1] master.RegionStates(264): Region {NAME => 'testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,j,1365336962398.38f0eb146fb473b19541deaada3419b9.', STARTKEY => 'j', ENDKEY => '', ENCODED => 38f0eb146fb473b19541deaada3419b9,} transitioned from {testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,j,1365336962398.38f0eb146fb473b19541deaada3419b9. state=OFFLINE, ts=1365336962990, server=null} to {testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,j,1365336962398.38f0eb146fb473b19541deaada3419b9. state=OFFLINE, ts=1365336962996, server=null} 2013-04-07 12:16:02,996 DEBUG [asf001.sp2.ygridcore.net,33796,1365336878036-GeneralBulkAssigner-1] zookeeper.ZKAssign(177): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Async create of unassigned node for 38f0eb146fb473b19541deaada3419b9 with OFFLINE state 2013-04-07 12:16:03,001 DEBUG [Thread-200-EventThread] master.OfflineCallback(69): rs={testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,c,1365336962398.4432e800611c00a4ae168862a273fc70. state=OFFLINE, ts=1365336962989, server=null}, server=asf001.sp2.ygridcore.net,43802,1365336889418 2013-04-07 12:16:03,001 DEBUG [Thread-200-EventThread] master.OfflineCallback(69): rs={testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,e,1365336962398.dbb82f7f432d6664327da0b8f15b7d76. state=OFFLINE, ts=1365336962989, server=null}, server=asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:16:03,002 DEBUG [Thread-200-EventThread] master.OfflineCallback(69): rs={testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,,1365336962397.d4e129e51e67d798e17083623fdde133. state=OFFLINE, ts=1365336962989, server=null}, server=asf001.sp2.ygridcore.net,43802,1365336889418 2013-04-07 12:16:03,002 DEBUG [Thread-200-EventThread] master.OfflineCallback(69): rs={testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,f,1365336962398.eff8376859f5e6ba3201ec0ec9536d0f. state=OFFLINE, ts=1365336962989, server=null}, server=asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:16:03,003 DEBUG [Thread-200-EventThread] master.OfflineCallback(69): rs={testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,g,1365336962398.682475e1358b7306bd3406e9330263a4. state=OFFLINE, ts=1365336962989, server=null}, server=asf001.sp2.ygridcore.net,49518,1365336962323 2013-04-07 12:16:03,003 DEBUG [Thread-200-EventThread] zookeeper.ZooKeeperWatcher(274): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/region-in-transition 2013-04-07 12:16:03,004 DEBUG [Thread-200-EventThread] master.OfflineCallback(69): rs={testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,i,1365336962398.385e5846630b0cb4493758961a2c2457. state=OFFLINE, ts=1365336962989, server=null}, server=asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:16:03,005 DEBUG [Thread-200-EventThread] zookeeper.ZooKeeperWatcher(274): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/region-in-transition 2013-04-07 12:16:03,005 DEBUG [Thread-200-EventThread] master.OfflineCallback(69): rs={testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,h,1365336962398.1ce6d62427641e2a0e135d3d27b54b5d. state=OFFLINE, ts=1365336962989, server=null}, server=asf001.sp2.ygridcore.net,43802,1365336889418 2013-04-07 12:16:03,006 DEBUG [Thread-200-EventThread] master.OfflineCallback(69): rs={testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,a,1365336962398.b1b340dff1e06fd6cb0b0a9c64964551. state=OFFLINE, ts=1365336962989, server=null}, server=asf001.sp2.ygridcore.net,49518,1365336962323 2013-04-07 12:16:03,006 DEBUG [Thread-200-EventThread] master.OfflineCallback(69): rs={testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,b,1365336962398.11c8123e8d563f264d947ffa106e4c7c. state=OFFLINE, ts=1365336962989, server=null}, server=asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:16:03,007 DEBUG [Thread-200-EventThread] master.OfflineCallback$ExistCallback(106): rs={testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,d,1365336962398.e99798d2c4d566ddb063efe03842457f. state=OFFLINE, ts=1365336962989, server=null}, server=asf001.sp2.ygridcore.net,49518,1365336962323 2013-04-07 12:16:03,007 DEBUG [Thread-200-EventThread] master.OfflineCallback(69): rs={testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,j,1365336962398.38f0eb146fb473b19541deaada3419b9. state=OFFLINE, ts=1365336962990, server=null}, server=asf001.sp2.ygridcore.net,49518,1365336962323 2013-04-07 12:16:03,007 INFO [asf001.sp2.ygridcore.net,33796,1365336878036-GeneralBulkAssigner-1] master.AssignmentManager(1482): asf001.sp2.ygridcore.net,49518,1365336962323 unassigned znodes=1 of total=4 2013-04-07 12:16:03,008 DEBUG [Thread-200-EventThread] master.OfflineCallback$ExistCallback(106): rs={testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,c,1365336962398.4432e800611c00a4ae168862a273fc70. state=OFFLINE, ts=1365336962989, server=null}, server=asf001.sp2.ygridcore.net,43802,1365336889418 2013-04-07 12:16:03,008 DEBUG [Thread-200-EventThread] master.OfflineCallback$ExistCallback(106): rs={testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,e,1365336962398.dbb82f7f432d6664327da0b8f15b7d76. state=OFFLINE, ts=1365336962989, server=null}, server=asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:16:03,008 DEBUG [Thread-200-EventThread] master.OfflineCallback$ExistCallback(106): rs={testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,,1365336962397.d4e129e51e67d798e17083623fdde133. state=OFFLINE, ts=1365336962989, server=null}, server=asf001.sp2.ygridcore.net,43802,1365336889418 2013-04-07 12:16:03,009 DEBUG [Thread-200-EventThread] master.OfflineCallback$ExistCallback(106): rs={testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,f,1365336962398.eff8376859f5e6ba3201ec0ec9536d0f. state=OFFLINE, ts=1365336962989, server=null}, server=asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:16:03,009 DEBUG [Thread-200-EventThread] master.OfflineCallback$ExistCallback(106): rs={testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,g,1365336962398.682475e1358b7306bd3406e9330263a4. state=OFFLINE, ts=1365336962989, server=null}, server=asf001.sp2.ygridcore.net,49518,1365336962323 2013-04-07 12:16:03,010 DEBUG [hbase-am-zkevent-worker-pool-2-thread-21] zookeeper.ZKUtil(1680): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Retrieved 155 byte(s) of data from znode /hbase/region-in-transition/d4e129e51e67d798e17083623fdde133 and set watcher; region=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,,1365336962397.d4e129e51e67d798e17083623fdde133., state=M_ZK_REGION_OFFLINE, servername=asf001.sp2.ygridcore.net,43802,1365336889418, createTime=1365336962994, payload.length=0 2013-04-07 12:16:03,010 INFO [asf001.sp2.ygridcore.net,33796,1365336878036-GeneralBulkAssigner-0] master.AssignmentManager(1482): asf001.sp2.ygridcore.net,43802,1365336889418 unassigned znodes=2 of total=3 2013-04-07 12:16:03,011 DEBUG [Thread-200-EventThread] master.OfflineCallback$ExistCallback(106): rs={testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,i,1365336962398.385e5846630b0cb4493758961a2c2457. state=OFFLINE, ts=1365336962989, server=null}, server=asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:16:03,012 INFO [asf001.sp2.ygridcore.net,33796,1365336878036-GeneralBulkAssigner-2] master.AssignmentManager(1482): asf001.sp2.ygridcore.net,60884,1365336878389 unassigned znodes=3 of total=4 2013-04-07 12:16:03,012 DEBUG [Thread-200-EventThread] master.OfflineCallback$ExistCallback(106): rs={testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,h,1365336962398.1ce6d62427641e2a0e135d3d27b54b5d. state=OFFLINE, ts=1365336962989, server=null}, server=asf001.sp2.ygridcore.net,43802,1365336889418 2013-04-07 12:16:03,012 DEBUG [Thread-200-EventThread] master.OfflineCallback$ExistCallback(106): rs={testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,a,1365336962398.b1b340dff1e06fd6cb0b0a9c64964551. state=OFFLINE, ts=1365336962989, server=null}, server=asf001.sp2.ygridcore.net,49518,1365336962323 2013-04-07 12:16:03,013 DEBUG [Thread-200-EventThread] master.OfflineCallback$ExistCallback(106): rs={testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,b,1365336962398.11c8123e8d563f264d947ffa106e4c7c. state=OFFLINE, ts=1365336962989, server=null}, server=asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:16:03,013 INFO [asf001.sp2.ygridcore.net,33796,1365336878036-GeneralBulkAssigner-1] master.AssignmentManager(1482): asf001.sp2.ygridcore.net,49518,1365336962323 unassigned znodes=3 of total=4 2013-04-07 12:16:03,013 DEBUG [Thread-200-EventThread] master.OfflineCallback$ExistCallback(106): rs={testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,j,1365336962398.38f0eb146fb473b19541deaada3419b9. state=OFFLINE, ts=1365336962990, server=null}, server=asf001.sp2.ygridcore.net,49518,1365336962323 2013-04-07 12:16:03,013 DEBUG [hbase-am-zkevent-worker-pool-2-thread-21] zookeeper.ZKUtil(1680): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Retrieved 156 byte(s) of data from znode /hbase/region-in-transition/4432e800611c00a4ae168862a273fc70 and set watcher; region=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,c,1365336962398.4432e800611c00a4ae168862a273fc70., state=M_ZK_REGION_OFFLINE, servername=asf001.sp2.ygridcore.net,43802,1365336889418, createTime=1365336962993, payload.length=0 2013-04-07 12:16:03,014 DEBUG [hbase-am-zkevent-worker-pool-2-thread-22] zookeeper.ZKUtil(1680): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Retrieved 156 byte(s) of data from znode /hbase/region-in-transition/11c8123e8d563f264d947ffa106e4c7c and set watcher; region=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,b,1365336962398.11c8123e8d563f264d947ffa106e4c7c., state=M_ZK_REGION_OFFLINE, servername=asf001.sp2.ygridcore.net,60884,1365336878389, createTime=1365336962996, payload.length=0 2013-04-07 12:16:03,014 DEBUG [hbase-am-zkevent-worker-pool-2-thread-23] zookeeper.ZKUtil(1680): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Retrieved 156 byte(s) of data from znode /hbase/region-in-transition/11c8123e8d563f264d947ffa106e4c7c and set watcher; region=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,b,1365336962398.11c8123e8d563f264d947ffa106e4c7c., state=M_ZK_REGION_OFFLINE, servername=asf001.sp2.ygridcore.net,60884,1365336878389, createTime=1365336962996, payload.length=0 2013-04-07 12:16:03,015 DEBUG [hbase-am-zkevent-worker-pool-2-thread-21] zookeeper.ZKUtil(1680): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Retrieved 156 byte(s) of data from znode /hbase/region-in-transition/dbb82f7f432d6664327da0b8f15b7d76 and set watcher; region=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,e,1365336962398.dbb82f7f432d6664327da0b8f15b7d76., state=M_ZK_REGION_OFFLINE, servername=asf001.sp2.ygridcore.net,60884,1365336878389, createTime=1365336962994, payload.length=0 2013-04-07 12:16:03,015 DEBUG [hbase-am-zkevent-worker-pool-2-thread-22] zookeeper.ZKUtil(1680): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Retrieved 156 byte(s) of data from znode /hbase/region-in-transition/38f0eb146fb473b19541deaada3419b9 and set watcher; region=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,j,1365336962398.38f0eb146fb473b19541deaada3419b9., state=M_ZK_REGION_OFFLINE, servername=asf001.sp2.ygridcore.net,49518,1365336962323, createTime=1365336962997, payload.length=0 2013-04-07 12:16:03,015 DEBUG [hbase-am-zkevent-worker-pool-2-thread-23] zookeeper.ZKUtil(1680): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Retrieved 156 byte(s) of data from znode /hbase/region-in-transition/38f0eb146fb473b19541deaada3419b9 and set watcher; region=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,j,1365336962398.38f0eb146fb473b19541deaada3419b9., state=M_ZK_REGION_OFFLINE, servername=asf001.sp2.ygridcore.net,49518,1365336962323, createTime=1365336962997, payload.length=0 2013-04-07 12:16:03,016 INFO [asf001.sp2.ygridcore.net,33796,1365336878036-GeneralBulkAssigner-0] master.AssignmentManager(1482): asf001.sp2.ygridcore.net,43802,1365336889418 unassigned znodes=3 of total=3 2013-04-07 12:16:03,016 INFO [asf001.sp2.ygridcore.net,33796,1365336878036-GeneralBulkAssigner-0] master.RegionStates(264): Region {NAME => 'testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,c,1365336962398.4432e800611c00a4ae168862a273fc70.', STARTKEY => 'c', ENDKEY => 'd', ENCODED => 4432e800611c00a4ae168862a273fc70,} transitioned from {testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,c,1365336962398.4432e800611c00a4ae168862a273fc70. state=OFFLINE, ts=1365336962992, server=null} to {testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,c,1365336962398.4432e800611c00a4ae168862a273fc70. state=PENDING_OPEN, ts=1365336963016, server=asf001.sp2.ygridcore.net,43802,1365336889418} 2013-04-07 12:16:03,016 DEBUG [hbase-am-zkevent-worker-pool-2-thread-21] zookeeper.ZKUtil(1680): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Retrieved 156 byte(s) of data from znode /hbase/region-in-transition/682475e1358b7306bd3406e9330263a4 and set watcher; region=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,g,1365336962398.682475e1358b7306bd3406e9330263a4., state=M_ZK_REGION_OFFLINE, servername=asf001.sp2.ygridcore.net,49518,1365336962323, createTime=1365336962995, payload.length=0 2013-04-07 12:16:03,016 INFO [asf001.sp2.ygridcore.net,33796,1365336878036-GeneralBulkAssigner-0] master.RegionStates(264): Region {NAME => 'testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,,1365336962397.d4e129e51e67d798e17083623fdde133.', STARTKEY => '', ENDKEY => 'a', ENCODED => d4e129e51e67d798e17083623fdde133,} transitioned from {testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,,1365336962397.d4e129e51e67d798e17083623fdde133. state=OFFLINE, ts=1365336962994, server=null} to {testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,,1365336962397.d4e129e51e67d798e17083623fdde133. state=PENDING_OPEN, ts=1365336963016, server=asf001.sp2.ygridcore.net,43802,1365336889418} 2013-04-07 12:16:03,016 DEBUG [hbase-am-zkevent-worker-pool-2-thread-22] zookeeper.ZKUtil(1680): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Retrieved 156 byte(s) of data from znode /hbase/region-in-transition/682475e1358b7306bd3406e9330263a4 and set watcher; region=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,g,1365336962398.682475e1358b7306bd3406e9330263a4., state=M_ZK_REGION_OFFLINE, servername=asf001.sp2.ygridcore.net,49518,1365336962323, createTime=1365336962995, payload.length=0 2013-04-07 12:16:03,016 INFO [asf001.sp2.ygridcore.net,33796,1365336878036-GeneralBulkAssigner-0] master.RegionStates(264): Region {NAME => 'testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,h,1365336962398.1ce6d62427641e2a0e135d3d27b54b5d.', STARTKEY => 'h', ENDKEY => 'i', ENCODED => 1ce6d62427641e2a0e135d3d27b54b5d,} transitioned from {testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,h,1365336962398.1ce6d62427641e2a0e135d3d27b54b5d. state=OFFLINE, ts=1365336962994, server=null} to {testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,h,1365336962398.1ce6d62427641e2a0e135d3d27b54b5d. state=PENDING_OPEN, ts=1365336963016, server=asf001.sp2.ygridcore.net,43802,1365336889418} 2013-04-07 12:16:03,017 INFO [asf001.sp2.ygridcore.net,33796,1365336878036-GeneralBulkAssigner-2] master.AssignmentManager(1482): asf001.sp2.ygridcore.net,60884,1365336878389 unassigned znodes=4 of total=4 2013-04-07 12:16:03,018 INFO [asf001.sp2.ygridcore.net,33796,1365336878036-GeneralBulkAssigner-2] master.RegionStates(264): Region {NAME => 'testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,e,1365336962398.dbb82f7f432d6664327da0b8f15b7d76.', STARTKEY => 'e', ENDKEY => 'f', ENCODED => dbb82f7f432d6664327da0b8f15b7d76,} transitioned from {testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,e,1365336962398.dbb82f7f432d6664327da0b8f15b7d76. state=OFFLINE, ts=1365336962993, server=null} to {testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,e,1365336962398.dbb82f7f432d6664327da0b8f15b7d76. state=PENDING_OPEN, ts=1365336963018, server=asf001.sp2.ygridcore.net,60884,1365336878389} 2013-04-07 12:16:03,018 INFO [asf001.sp2.ygridcore.net,33796,1365336878036-GeneralBulkAssigner-2] master.RegionStates(264): Region {NAME => 'testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,f,1365336962398.eff8376859f5e6ba3201ec0ec9536d0f.', STARTKEY => 'f', ENDKEY => 'g', ENCODED => eff8376859f5e6ba3201ec0ec9536d0f,} transitioned from {testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,f,1365336962398.eff8376859f5e6ba3201ec0ec9536d0f. state=OFFLINE, ts=1365336962994, server=null} to {testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,f,1365336962398.eff8376859f5e6ba3201ec0ec9536d0f. state=PENDING_OPEN, ts=1365336963018, server=asf001.sp2.ygridcore.net,60884,1365336878389} 2013-04-07 12:16:03,018 INFO [asf001.sp2.ygridcore.net,33796,1365336878036-GeneralBulkAssigner-2] master.RegionStates(264): Region {NAME => 'testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,i,1365336962398.385e5846630b0cb4493758961a2c2457.', STARTKEY => 'i', ENDKEY => 'j', ENCODED => 385e5846630b0cb4493758961a2c2457,} transitioned from {testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,i,1365336962398.385e5846630b0cb4493758961a2c2457. state=OFFLINE, ts=1365336962995, server=null} to {testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,i,1365336962398.385e5846630b0cb4493758961a2c2457. state=PENDING_OPEN, ts=1365336963018, server=asf001.sp2.ygridcore.net,60884,1365336878389} 2013-04-07 12:16:03,018 DEBUG [hbase-am-zkevent-worker-pool-2-thread-21] zookeeper.ZKUtil(1680): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Retrieved 156 byte(s) of data from znode /hbase/region-in-transition/eff8376859f5e6ba3201ec0ec9536d0f and set watcher; region=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,f,1365336962398.eff8376859f5e6ba3201ec0ec9536d0f., state=M_ZK_REGION_OFFLINE, servername=asf001.sp2.ygridcore.net,60884,1365336878389, createTime=1365336962994, payload.length=0 2013-04-07 12:16:03,018 INFO [asf001.sp2.ygridcore.net,33796,1365336878036-GeneralBulkAssigner-2] master.RegionStates(264): Region {NAME => 'testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,b,1365336962398.11c8123e8d563f264d947ffa106e4c7c.', STARTKEY => 'b', ENDKEY => 'c', ENCODED => 11c8123e8d563f264d947ffa106e4c7c,} transitioned from {testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,b,1365336962398.11c8123e8d563f264d947ffa106e4c7c. state=OFFLINE, ts=1365336962995, server=null} to {testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,b,1365336962398.11c8123e8d563f264d947ffa106e4c7c. state=PENDING_OPEN, ts=1365336963018, server=asf001.sp2.ygridcore.net,60884,1365336878389} 2013-04-07 12:16:03,021 DEBUG [hbase-am-zkevent-worker-pool-2-thread-23] zookeeper.ZKUtil(1680): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Retrieved 156 byte(s) of data from znode /hbase/region-in-transition/682475e1358b7306bd3406e9330263a4 and set watcher; region=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,g,1365336962398.682475e1358b7306bd3406e9330263a4., state=M_ZK_REGION_OFFLINE, servername=asf001.sp2.ygridcore.net,49518,1365336962323, createTime=1365336962995, payload.length=0 2013-04-07 12:16:03,022 DEBUG [hbase-am-zkevent-worker-pool-2-thread-22] zookeeper.ZKUtil(1680): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Retrieved 156 byte(s) of data from znode /hbase/region-in-transition/1ce6d62427641e2a0e135d3d27b54b5d and set watcher; region=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,h,1365336962398.1ce6d62427641e2a0e135d3d27b54b5d., state=M_ZK_REGION_OFFLINE, servername=asf001.sp2.ygridcore.net,43802,1365336889418, createTime=1365336962995, payload.length=0 2013-04-07 12:16:03,022 INFO [PRI IPC Server handler 6 on 60884] regionserver.HRegionServer(3346): Received request to open region: testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,e,1365336962398.dbb82f7f432d6664327da0b8f15b7d76. on asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:16:03,025 DEBUG [hbase-am-zkevent-worker-pool-2-thread-21] zookeeper.ZKUtil(1680): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Retrieved 156 byte(s) of data from znode /hbase/region-in-transition/e99798d2c4d566ddb063efe03842457f and set watcher; region=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,d,1365336962398.e99798d2c4d566ddb063efe03842457f., state=M_ZK_REGION_OFFLINE, servername=asf001.sp2.ygridcore.net,49518,1365336962323, createTime=1365336962993, payload.length=0 2013-04-07 12:16:03,026 DEBUG [hbase-am-zkevent-worker-pool-2-thread-23] zookeeper.ZKUtil(1680): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Retrieved 156 byte(s) of data from znode /hbase/region-in-transition/1ce6d62427641e2a0e135d3d27b54b5d and set watcher; region=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,h,1365336962398.1ce6d62427641e2a0e135d3d27b54b5d., state=M_ZK_REGION_OFFLINE, servername=asf001.sp2.ygridcore.net,43802,1365336889418, createTime=1365336962995, payload.length=0 2013-04-07 12:16:03,026 DEBUG [hbase-am-zkevent-worker-pool-2-thread-22] zookeeper.ZKUtil(1680): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Retrieved 156 byte(s) of data from znode /hbase/region-in-transition/4432e800611c00a4ae168862a273fc70 and set watcher; region=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,c,1365336962398.4432e800611c00a4ae168862a273fc70., state=M_ZK_REGION_OFFLINE, servername=asf001.sp2.ygridcore.net,43802,1365336889418, createTime=1365336962993, payload.length=0 2013-04-07 12:16:03,029 DEBUG [hbase-am-zkevent-worker-pool-2-thread-23] zookeeper.ZKUtil(1680): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Retrieved 156 byte(s) of data from znode /hbase/region-in-transition/4432e800611c00a4ae168862a273fc70 and set watcher; region=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,c,1365336962398.4432e800611c00a4ae168862a273fc70., state=M_ZK_REGION_OFFLINE, servername=asf001.sp2.ygridcore.net,43802,1365336889418, createTime=1365336962993, payload.length=0 2013-04-07 12:16:03,030 DEBUG [hbase-am-zkevent-worker-pool-2-thread-22] zookeeper.ZKUtil(1680): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Retrieved 155 byte(s) of data from znode /hbase/region-in-transition/d4e129e51e67d798e17083623fdde133 and set watcher; region=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,,1365336962397.d4e129e51e67d798e17083623fdde133., state=M_ZK_REGION_OFFLINE, servername=asf001.sp2.ygridcore.net,43802,1365336889418, createTime=1365336962994, payload.length=0 2013-04-07 12:16:03,033 DEBUG [hbase-am-zkevent-worker-pool-2-thread-23] zookeeper.ZKUtil(1680): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Retrieved 155 byte(s) of data from znode /hbase/region-in-transition/d4e129e51e67d798e17083623fdde133 and set watcher; region=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,,1365336962397.d4e129e51e67d798e17083623fdde133., state=M_ZK_REGION_OFFLINE, servername=asf001.sp2.ygridcore.net,43802,1365336889418, createTime=1365336962994, payload.length=0 2013-04-07 12:16:03,034 DEBUG [hbase-am-zkevent-worker-pool-2-thread-22] zookeeper.ZKUtil(1680): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Retrieved 156 byte(s) of data from znode /hbase/region-in-transition/dbb82f7f432d6664327da0b8f15b7d76 and set watcher; region=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,e,1365336962398.dbb82f7f432d6664327da0b8f15b7d76., state=M_ZK_REGION_OFFLINE, servername=asf001.sp2.ygridcore.net,60884,1365336878389, createTime=1365336962994, payload.length=0 2013-04-07 12:16:03,035 DEBUG [hbase-am-zkevent-worker-pool-2-thread-23] zookeeper.ZKUtil(1680): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Retrieved 156 byte(s) of data from znode /hbase/region-in-transition/dbb82f7f432d6664327da0b8f15b7d76 and set watcher; region=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,e,1365336962398.dbb82f7f432d6664327da0b8f15b7d76., state=M_ZK_REGION_OFFLINE, servername=asf001.sp2.ygridcore.net,60884,1365336878389, createTime=1365336962994, payload.length=0 2013-04-07 12:16:03,038 DEBUG [hbase-am-zkevent-worker-pool-2-thread-22] zookeeper.ZKUtil(1680): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Retrieved 156 byte(s) of data from znode /hbase/region-in-transition/eff8376859f5e6ba3201ec0ec9536d0f and set watcher; region=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,f,1365336962398.eff8376859f5e6ba3201ec0ec9536d0f., state=M_ZK_REGION_OFFLINE, servername=asf001.sp2.ygridcore.net,60884,1365336878389, createTime=1365336962994, payload.length=0 2013-04-07 12:16:03,039 INFO [PRI IPC Server handler 6 on 60884] regionserver.HRegionServer(3346): Received request to open region: testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,f,1365336962398.eff8376859f5e6ba3201ec0ec9536d0f. on asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:16:03,040 INFO [PRI IPC Server handler 6 on 60884] regionserver.HRegionServer(3346): Received request to open region: testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,i,1365336962398.385e5846630b0cb4493758961a2c2457. on asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:16:03,040 INFO [PRI IPC Server handler 6 on 60884] regionserver.HRegionServer(3346): Received request to open region: testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,b,1365336962398.11c8123e8d563f264d947ffa106e4c7c. on asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:16:03,045 DEBUG [hbase-am-zkevent-worker-pool-2-thread-23] zookeeper.ZKUtil(1680): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Retrieved 156 byte(s) of data from znode /hbase/region-in-transition/eff8376859f5e6ba3201ec0ec9536d0f and set watcher; region=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,f,1365336962398.eff8376859f5e6ba3201ec0ec9536d0f., state=M_ZK_REGION_OFFLINE, servername=asf001.sp2.ygridcore.net,60884,1365336878389, createTime=1365336962994, payload.length=0 2013-04-07 12:16:03,049 DEBUG [Thread-200-EventThread] zookeeper.ZooKeeperWatcher(274): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/region-in-transition/4432e800611c00a4ae168862a273fc70 2013-04-07 12:16:03,049 DEBUG [MASTER_TABLE_OPERATIONS-asf001.sp2.ygridcore.net,33796,1365336878036-0] master.GeneralBulkAssigner(153): bulk assigning total 11 regions to 3 servers, took 57ms, with 11 regions still in transition 2013-04-07 12:16:03,050 INFO [MASTER_TABLE_OPERATIONS-asf001.sp2.ygridcore.net,33796,1365336878036-0] master.AssignmentManager(2377): Bulk assigning done 2013-04-07 12:16:03,055 DEBUG [hbase-am-zkevent-worker-pool-2-thread-22] zookeeper.ZKUtil(1680): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Retrieved 156 byte(s) of data from znode /hbase/region-in-transition/385e5846630b0cb4493758961a2c2457 and set watcher; region=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,i,1365336962398.385e5846630b0cb4493758961a2c2457., state=M_ZK_REGION_OFFLINE, servername=asf001.sp2.ygridcore.net,60884,1365336878389, createTime=1365336962995, payload.length=0 2013-04-07 12:16:03,056 DEBUG [RS_OPEN_REGION-asf001.sp2.ygridcore.net,60884,1365336878389-1] zookeeper.ZKUtil(1680): regionserver:60884-0x13de46b5f8b0001 Retrieved 156 byte(s) of data from znode /hbase/region-in-transition/eff8376859f5e6ba3201ec0ec9536d0f; data=region=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,f,1365336962398.eff8376859f5e6ba3201ec0ec9536d0f., state=M_ZK_REGION_OFFLINE, servername=asf001.sp2.ygridcore.net,60884,1365336878389, createTime=1365336962994, payload.length=0 2013-04-07 12:16:03,059 DEBUG [hbase-am-zkevent-worker-pool-2-thread-23] zookeeper.ZKUtil(1680): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Retrieved 156 byte(s) of data from znode /hbase/region-in-transition/385e5846630b0cb4493758961a2c2457 and set watcher; region=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,i,1365336962398.385e5846630b0cb4493758961a2c2457., state=M_ZK_REGION_OFFLINE, servername=asf001.sp2.ygridcore.net,60884,1365336878389, createTime=1365336962995, payload.length=0 2013-04-07 12:16:03,059 DEBUG [Thread-200-EventThread] zookeeper.ZooKeeperWatcher(274): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/region-in-transition/1ce6d62427641e2a0e135d3d27b54b5d 2013-04-07 12:16:03,059 DEBUG [RS_OPEN_REGION-asf001.sp2.ygridcore.net,60884,1365336878389-0] regionserver.MetricsRegionSourceImpl(54): Creating new MetricsRegionSourceImpl for table testRegionAssignmentAfterMasterRecoveryDueToZKExpiry dbb82f7f432d6664327da0b8f15b7d76 2013-04-07 12:16:03,060 DEBUG [RS_OPEN_REGION-asf001.sp2.ygridcore.net,60884,1365336878389-0] regionserver.HRegion(496): Instantiated testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,e,1365336962398.dbb82f7f432d6664327da0b8f15b7d76. 2013-04-07 12:16:03,061 DEBUG [Thread-200-EventThread] zookeeper.ZooKeeperWatcher(274): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/region-in-transition/dbb82f7f432d6664327da0b8f15b7d76 2013-04-07 12:16:03,062 DEBUG [Thread-200-EventThread] zookeeper.ZooKeeperWatcher(274): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/region-in-transition/e99798d2c4d566ddb063efe03842457f 2013-04-07 12:16:03,064 DEBUG [Thread-200-EventThread] zookeeper.ZooKeeperWatcher(274): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/region-in-transition/682475e1358b7306bd3406e9330263a4 2013-04-07 12:16:03,064 DEBUG [Thread-200-EventThread] zookeeper.ZooKeeperWatcher(274): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/region-in-transition/d4e129e51e67d798e17083623fdde133 2013-04-07 12:16:03,068 DEBUG [Thread-200-EventThread] zookeeper.ZooKeeperWatcher(274): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/region-in-transition/eff8376859f5e6ba3201ec0ec9536d0f 2013-04-07 12:16:03,068 DEBUG [RS_OPEN_REGION-asf001.sp2.ygridcore.net,60884,1365336878389-1] zookeeper.ZKAssign(784): regionserver:60884-0x13de46b5f8b0001 Successfully transitioned node eff8376859f5e6ba3201ec0ec9536d0f from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING 2013-04-07 12:16:03,068 INFO [RS_OPEN_REGION-asf001.sp2.ygridcore.net,60884,1365336878389-1] regionserver.HRegion(4023): HRegion.openHRegion Region name ==testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,f,1365336962398.eff8376859f5e6ba3201ec0ec9536d0f. 2013-04-07 12:16:03,069 DEBUG [RS_OPEN_REGION-asf001.sp2.ygridcore.net,60884,1365336878389-1] regionserver.HRegion(4025): Opening region: {NAME => 'testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,f,1365336962398.eff8376859f5e6ba3201ec0ec9536d0f.', STARTKEY => 'f', ENDKEY => 'g', ENCODED => eff8376859f5e6ba3201ec0ec9536d0f,} 2013-04-07 12:16:03,070 DEBUG [hbase-am-zkevent-worker-pool-20-thread-24] zookeeper.ZKUtil(1680): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Retrieved 156 byte(s) of data from znode /hbase/region-in-transition/4432e800611c00a4ae168862a273fc70 and set watcher; region=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,c,1365336962398.4432e800611c00a4ae168862a273fc70., state=RS_ZK_REGION_OPENING, servername=asf001.sp2.ygridcore.net,43802,1365336889418, createTime=1365336963040, payload.length=0 2013-04-07 12:16:03,070 DEBUG [hbase-am-zkevent-worker-pool-20-thread-24] master.AssignmentManager(740): Handling transition=RS_ZK_REGION_OPENING, server=asf001.sp2.ygridcore.net,43802,1365336889418, region=4432e800611c00a4ae168862a273fc70, current state from region state map ={testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,c,1365336962398.4432e800611c00a4ae168862a273fc70. state=PENDING_OPEN, ts=1365336963016, server=asf001.sp2.ygridcore.net,43802,1365336889418} 2013-04-07 12:16:03,070 INFO [hbase-am-zkevent-worker-pool-20-thread-24] master.RegionStates(264): Region {NAME => 'testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,c,1365336962398.4432e800611c00a4ae168862a273fc70.', STARTKEY => 'c', ENDKEY => 'd', ENCODED => 4432e800611c00a4ae168862a273fc70,} transitioned from {testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,c,1365336962398.4432e800611c00a4ae168862a273fc70. state=PENDING_OPEN, ts=1365336963016, server=asf001.sp2.ygridcore.net,43802,1365336889418} to {testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,c,1365336962398.4432e800611c00a4ae168862a273fc70. state=OPENING, ts=1365336963070, server=asf001.sp2.ygridcore.net,43802,1365336889418} 2013-04-07 12:16:03,070 DEBUG [RS_OPEN_REGION-asf001.sp2.ygridcore.net,60884,1365336878389-1] regionserver.HRegion(496): Instantiated testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,f,1365336962398.eff8376859f5e6ba3201ec0ec9536d0f. 2013-04-07 12:16:03,071 DEBUG [RS_OPEN_REGION-asf001.sp2.ygridcore.net,60884,1365336878389-2] zookeeper.ZKAssign(784): regionserver:60884-0x13de46b5f8b0001 Successfully transitioned node 385e5846630b0cb4493758961a2c2457 from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING 2013-04-07 12:16:03,071 DEBUG [Thread-200-EventThread] zookeeper.ZooKeeperWatcher(274): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/region-in-transition/385e5846630b0cb4493758961a2c2457 2013-04-07 12:16:03,072 DEBUG [hbase-am-zkevent-worker-pool-2-thread-24] zookeeper.ZKUtil(1680): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Retrieved 156 byte(s) of data from znode /hbase/region-in-transition/4432e800611c00a4ae168862a273fc70 and set watcher; region=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,c,1365336962398.4432e800611c00a4ae168862a273fc70., state=RS_ZK_REGION_OPENING, servername=asf001.sp2.ygridcore.net,43802,1365336889418, createTime=1365336963040, payload.length=0 2013-04-07 12:16:03,072 DEBUG [hbase-am-zkevent-worker-pool-2-thread-24] master.AssignmentManager(740): Handling transition=RS_ZK_REGION_OPENING, server=asf001.sp2.ygridcore.net,43802,1365336889418, region=4432e800611c00a4ae168862a273fc70, current state from region state map =null 2013-04-07 12:16:03,072 DEBUG [hbase-am-zkevent-worker-pool-2-thread-22] zookeeper.ZKUtil(1680): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Retrieved 156 byte(s) of data from znode /hbase/region-in-transition/b1b340dff1e06fd6cb0b0a9c64964551 and set watcher; region=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,a,1365336962398.b1b340dff1e06fd6cb0b0a9c64964551., state=M_ZK_REGION_OFFLINE, servername=asf001.sp2.ygridcore.net,49518,1365336962323, createTime=1365336962995, payload.length=0 2013-04-07 12:16:03,072 DEBUG [hbase-am-zkevent-worker-pool-2-thread-23] zookeeper.ZKUtil(1680): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Retrieved 156 byte(s) of data from znode /hbase/region-in-transition/b1b340dff1e06fd6cb0b0a9c64964551 and set watcher; region=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,a,1365336962398.b1b340dff1e06fd6cb0b0a9c64964551., state=M_ZK_REGION_OFFLINE, servername=asf001.sp2.ygridcore.net,49518,1365336962323, createTime=1365336962995, payload.length=0 2013-04-07 12:16:03,073 DEBUG [hbase-am-zkevent-worker-pool-20-thread-25] zookeeper.ZKUtil(1680): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Retrieved 156 byte(s) of data from znode /hbase/region-in-transition/1ce6d62427641e2a0e135d3d27b54b5d and set watcher; region=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,h,1365336962398.1ce6d62427641e2a0e135d3d27b54b5d., state=RS_ZK_REGION_OPENING, servername=asf001.sp2.ygridcore.net,43802,1365336889418, createTime=1365336963044, payload.length=0 2013-04-07 12:16:03,073 DEBUG [RS_OPEN_REGION-asf001.sp2.ygridcore.net,60884,1365336878389-2] regionserver.MetricsRegionSourceImpl(54): Creating new MetricsRegionSourceImpl for table testRegionAssignmentAfterMasterRecoveryDueToZKExpiry 385e5846630b0cb4493758961a2c2457 2013-04-07 12:16:03,073 DEBUG [hbase-am-zkevent-worker-pool-20-thread-25] master.AssignmentManager(740): Handling transition=RS_ZK_REGION_OPENING, server=asf001.sp2.ygridcore.net,43802,1365336889418, region=1ce6d62427641e2a0e135d3d27b54b5d, current state from region state map ={testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,h,1365336962398.1ce6d62427641e2a0e135d3d27b54b5d. state=PENDING_OPEN, ts=1365336963016, server=asf001.sp2.ygridcore.net,43802,1365336889418} 2013-04-07 12:16:03,073 DEBUG [RS_OPEN_REGION-asf001.sp2.ygridcore.net,60884,1365336878389-2] regionserver.HRegion(496): Instantiated testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,i,1365336962398.385e5846630b0cb4493758961a2c2457. 2013-04-07 12:16:03,073 INFO [hbase-am-zkevent-worker-pool-20-thread-25] master.RegionStates(264): Region {NAME => 'testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,h,1365336962398.1ce6d62427641e2a0e135d3d27b54b5d.', STARTKEY => 'h', ENDKEY => 'i', ENCODED => 1ce6d62427641e2a0e135d3d27b54b5d,} transitioned from {testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,h,1365336962398.1ce6d62427641e2a0e135d3d27b54b5d. state=PENDING_OPEN, ts=1365336963016, server=asf001.sp2.ygridcore.net,43802,1365336889418} to {testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,h,1365336962398.1ce6d62427641e2a0e135d3d27b54b5d. state=OPENING, ts=1365336963073, server=asf001.sp2.ygridcore.net,43802,1365336889418} 2013-04-07 12:16:03,073 DEBUG [hbase-am-zkevent-worker-pool-2-thread-25] zookeeper.ZKUtil(1680): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Retrieved 156 byte(s) of data from znode /hbase/region-in-transition/1ce6d62427641e2a0e135d3d27b54b5d and set watcher; region=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,h,1365336962398.1ce6d62427641e2a0e135d3d27b54b5d., state=RS_ZK_REGION_OPENING, servername=asf001.sp2.ygridcore.net,43802,1365336889418, createTime=1365336963044, payload.length=0 2013-04-07 12:16:03,077 DEBUG [Thread-200-EventThread] zookeeper.ZooKeeperWatcher(274): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/region-in-transition/b1b340dff1e06fd6cb0b0a9c64964551 2013-04-07 12:16:03,077 DEBUG [hbase-am-zkevent-worker-pool-20-thread-27] zookeeper.ZKUtil(1680): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Retrieved 156 byte(s) of data from znode /hbase/region-in-transition/e99798d2c4d566ddb063efe03842457f and set watcher; region=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,d,1365336962398.e99798d2c4d566ddb063efe03842457f., state=RS_ZK_REGION_OPENING, servername=asf001.sp2.ygridcore.net,49518,1365336962323, createTime=1365336963053, payload.length=0 2013-04-07 12:16:03,077 DEBUG [hbase-am-zkevent-worker-pool-20-thread-27] master.AssignmentManager(740): Handling transition=RS_ZK_REGION_OPENING, server=asf001.sp2.ygridcore.net,49518,1365336962323, region=e99798d2c4d566ddb063efe03842457f, current state from region state map ={testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,d,1365336962398.e99798d2c4d566ddb063efe03842457f. state=PENDING_OPEN, ts=1365336963019, server=asf001.sp2.ygridcore.net,49518,1365336962323} 2013-04-07 12:16:03,078 INFO [hbase-am-zkevent-worker-pool-20-thread-27] master.RegionStates(264): Region {NAME => 'testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,d,1365336962398.e99798d2c4d566ddb063efe03842457f.', STARTKEY => 'd', ENDKEY => 'e', ENCODED => e99798d2c4d566ddb063efe03842457f,} transitioned from {testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,d,1365336962398.e99798d2c4d566ddb063efe03842457f. state=PENDING_OPEN, ts=1365336963019, server=asf001.sp2.ygridcore.net,49518,1365336962323} to {testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,d,1365336962398.e99798d2c4d566ddb063efe03842457f. state=OPENING, ts=1365336963078, server=asf001.sp2.ygridcore.net,49518,1365336962323} 2013-04-07 12:16:03,078 DEBUG [hbase-am-zkevent-worker-pool-2-thread-24] zookeeper.ZKUtil(1680): hconnection-0x1f327e-0x13de46b5f8b0003 Retrieved 43 byte(s) of data from znode /hbase/meta-region-server; data=asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:16:03,079 DEBUG [hbase-am-zkevent-worker-pool-2-thread-24] zookeeper.ZKUtil(1680): hconnection-0x1f327e-0x13de46b5f8b0003 Retrieved 43 byte(s) of data from znode /hbase/meta-region-server; data=asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:16:03,081 DEBUG [hbase-am-zkevent-worker-pool-20-thread-28] zookeeper.ZKUtil(1680): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Retrieved 156 byte(s) of data from znode /hbase/region-in-transition/682475e1358b7306bd3406e9330263a4 and set watcher; region=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,g,1365336962398.682475e1358b7306bd3406e9330263a4., state=RS_ZK_REGION_OPENING, servername=asf001.sp2.ygridcore.net,49518,1365336962323, createTime=1365336963053, payload.length=0 2013-04-07 12:16:03,081 INFO [hbase-am-zkevent-worker-pool-2-thread-24] master.RegionStates(264): Region {NAME => 'testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,c,1365336962398.4432e800611c00a4ae168862a273fc70.', STARTKEY => 'c', ENDKEY => 'd', ENCODED => 4432e800611c00a4ae168862a273fc70,} transitioned from {testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,c,1365336962398.4432e800611c00a4ae168862a273fc70. state=OFFLINE, ts=1365336963081, server=null} to {testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,c,1365336962398.4432e800611c00a4ae168862a273fc70. state=OPENING, ts=1365336963081, server=asf001.sp2.ygridcore.net,43802,1365336889418} 2013-04-07 12:16:03,082 DEBUG [hbase-am-zkevent-worker-pool-20-thread-28] master.AssignmentManager(740): Handling transition=RS_ZK_REGION_OPENING, server=asf001.sp2.ygridcore.net,49518,1365336962323, region=682475e1358b7306bd3406e9330263a4, current state from region state map ={testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,g,1365336962398.682475e1358b7306bd3406e9330263a4. state=PENDING_OPEN, ts=1365336963019, server=asf001.sp2.ygridcore.net,49518,1365336962323} 2013-04-07 12:16:03,082 DEBUG [hbase-am-zkevent-worker-pool-2-thread-25] master.AssignmentManager(740): Handling transition=RS_ZK_REGION_OPENING, server=asf001.sp2.ygridcore.net,43802,1365336889418, region=1ce6d62427641e2a0e135d3d27b54b5d, current state from region state map =null 2013-04-07 12:16:03,082 INFO [hbase-am-zkevent-worker-pool-20-thread-28] master.RegionStates(264): Region {NAME => 'testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,g,1365336962398.682475e1358b7306bd3406e9330263a4.', STARTKEY => 'g', ENDKEY => 'h', ENCODED => 682475e1358b7306bd3406e9330263a4,} transitioned from {testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,g,1365336962398.682475e1358b7306bd3406e9330263a4. state=PENDING_OPEN, ts=1365336963019, server=asf001.sp2.ygridcore.net,49518,1365336962323} to {testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,g,1365336962398.682475e1358b7306bd3406e9330263a4. state=OPENING, ts=1365336963082, server=asf001.sp2.ygridcore.net,49518,1365336962323} 2013-04-07 12:16:03,083 DEBUG [hbase-am-zkevent-worker-pool-2-thread-27] zookeeper.ZKUtil(1680): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Retrieved 156 byte(s) of data from znode /hbase/region-in-transition/e99798d2c4d566ddb063efe03842457f and set watcher; region=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,d,1365336962398.e99798d2c4d566ddb063efe03842457f., state=RS_ZK_REGION_OPENING, servername=asf001.sp2.ygridcore.net,49518,1365336962323, createTime=1365336963053, payload.length=0 2013-04-07 12:16:03,084 DEBUG [RS_OPEN_REGION-asf001.sp2.ygridcore.net,60884,1365336878389-0] zookeeper.ZKUtil(1680): regionserver:60884-0x13de46b5f8b0001 Retrieved 156 byte(s) of data from znode /hbase/region-in-transition/dbb82f7f432d6664327da0b8f15b7d76; data=region=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,e,1365336962398.dbb82f7f432d6664327da0b8f15b7d76., state=RS_ZK_REGION_OPENING, servername=asf001.sp2.ygridcore.net,60884,1365336878389, createTime=1365336963043, payload.length=0 2013-04-07 12:16:03,084 DEBUG [hbase-am-zkevent-worker-pool-2-thread-25] zookeeper.ZKUtil(1680): hconnection-0x1f327e-0x13de46b5f8b0003 Retrieved 43 byte(s) of data from znode /hbase/meta-region-server; data=asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:16:03,084 INFO [RS_OPEN_REGION-asf001.sp2.ygridcore.net,60884,1365336878389-1] regionserver.HRegion(586): Onlined testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,f,1365336962398.eff8376859f5e6ba3201ec0ec9536d0f.; next sequenceid=0 2013-04-07 12:16:03,085 DEBUG [RS_OPEN_REGION-asf001.sp2.ygridcore.net,60884,1365336878389-1] zookeeper.ZKAssign(721): regionserver:60884-0x13de46b5f8b0001 Attempting to transition node eff8376859f5e6ba3201ec0ec9536d0f from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENING 2013-04-07 12:16:03,087 DEBUG [hbase-am-zkevent-worker-pool-20-thread-26] zookeeper.ZKUtil(1680): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Retrieved 156 byte(s) of data from znode /hbase/region-in-transition/dbb82f7f432d6664327da0b8f15b7d76 and set watcher; region=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,e,1365336962398.dbb82f7f432d6664327da0b8f15b7d76., state=RS_ZK_REGION_OPENING, servername=asf001.sp2.ygridcore.net,60884,1365336878389, createTime=1365336963043, payload.length=0 2013-04-07 12:16:03,087 DEBUG [hbase-am-zkevent-worker-pool-20-thread-26] master.AssignmentManager(740): Handling transition=RS_ZK_REGION_OPENING, server=asf001.sp2.ygridcore.net,60884,1365336878389, region=dbb82f7f432d6664327da0b8f15b7d76, current state from region state map ={testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,e,1365336962398.dbb82f7f432d6664327da0b8f15b7d76. state=PENDING_OPEN, ts=1365336963018, server=asf001.sp2.ygridcore.net,60884,1365336878389} 2013-04-07 12:16:03,087 INFO [hbase-am-zkevent-worker-pool-20-thread-26] master.RegionStates(264): Region {NAME => 'testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,e,1365336962398.dbb82f7f432d6664327da0b8f15b7d76.', STARTKEY => 'e', ENDKEY => 'f', ENCODED => dbb82f7f432d6664327da0b8f15b7d76,} transitioned from {testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,e,1365336962398.dbb82f7f432d6664327da0b8f15b7d76. state=PENDING_OPEN, ts=1365336963018, server=asf001.sp2.ygridcore.net,60884,1365336878389} to {testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,e,1365336962398.dbb82f7f432d6664327da0b8f15b7d76. state=OPENING, ts=1365336963087, server=asf001.sp2.ygridcore.net,60884,1365336878389} 2013-04-07 12:16:03,087 DEBUG [hbase-am-zkevent-worker-pool-2-thread-25] zookeeper.ZKUtil(1680): hconnection-0x1f327e-0x13de46b5f8b0003 Retrieved 43 byte(s) of data from znode /hbase/meta-region-server; data=asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:16:03,089 INFO [hbase-am-zkevent-worker-pool-2-thread-25] master.RegionStates(264): Region {NAME => 'testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,h,1365336962398.1ce6d62427641e2a0e135d3d27b54b5d.', STARTKEY => 'h', ENDKEY => 'i', ENCODED => 1ce6d62427641e2a0e135d3d27b54b5d,} transitioned from {testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,h,1365336962398.1ce6d62427641e2a0e135d3d27b54b5d. state=OFFLINE, ts=1365336963089, server=null} to {testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,h,1365336962398.1ce6d62427641e2a0e135d3d27b54b5d. state=OPENING, ts=1365336963089, server=asf001.sp2.ygridcore.net,43802,1365336889418} 2013-04-07 12:16:03,089 DEBUG [hbase-am-zkevent-worker-pool-2-thread-27] master.AssignmentManager(740): Handling transition=RS_ZK_REGION_OPENING, server=asf001.sp2.ygridcore.net,49518,1365336962323, region=e99798d2c4d566ddb063efe03842457f, current state from region state map =null 2013-04-07 12:16:03,090 DEBUG [hbase-am-zkevent-worker-pool-2-thread-28] zookeeper.ZKUtil(1680): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Retrieved 156 byte(s) of data from znode /hbase/region-in-transition/682475e1358b7306bd3406e9330263a4 and set watcher; region=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,g,1365336962398.682475e1358b7306bd3406e9330263a4., state=RS_ZK_REGION_OPENING, servername=asf001.sp2.ygridcore.net,49518,1365336962323, createTime=1365336963053, payload.length=0 2013-04-07 12:16:03,090 DEBUG [RS_OPEN_REGION-asf001.sp2.ygridcore.net,60884,1365336878389-0] zookeeper.ZKAssign(784): regionserver:60884-0x13de46b5f8b0001 Successfully transitioned node dbb82f7f432d6664327da0b8f15b7d76 from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENING 2013-04-07 12:16:03,091 DEBUG [hbase-am-zkevent-worker-pool-2-thread-27] zookeeper.ZKUtil(1680): hconnection-0x1f327e-0x13de46b5f8b0003 Retrieved 43 byte(s) of data from znode /hbase/meta-region-server; data=asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:16:03,093 DEBUG [hbase-am-zkevent-worker-pool-2-thread-29] zookeeper.ZKUtil(1680): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Retrieved 155 byte(s) of data from znode /hbase/region-in-transition/d4e129e51e67d798e17083623fdde133 and set watcher; region=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,,1365336962397.d4e129e51e67d798e17083623fdde133., state=RS_ZK_REGION_OPENING, servername=asf001.sp2.ygridcore.net,43802,1365336889418, createTime=1365336963049, payload.length=0 2013-04-07 12:16:03,095 DEBUG [hbase-am-zkevent-worker-pool-2-thread-27] zookeeper.ZKUtil(1680): hconnection-0x1f327e-0x13de46b5f8b0003 Retrieved 43 byte(s) of data from znode /hbase/meta-region-server; data=asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:16:03,097 INFO [hbase-am-zkevent-worker-pool-2-thread-27] master.RegionStates(264): Region {NAME => 'testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,d,1365336962398.e99798d2c4d566ddb063efe03842457f.', STARTKEY => 'd', ENDKEY => 'e', ENCODED => e99798d2c4d566ddb063efe03842457f,} transitioned from {testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,d,1365336962398.e99798d2c4d566ddb063efe03842457f. state=OFFLINE, ts=1365336963097, server=null} to {testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,d,1365336962398.e99798d2c4d566ddb063efe03842457f. state=OPENING, ts=1365336963097, server=asf001.sp2.ygridcore.net,49518,1365336962323} 2013-04-07 12:16:03,097 DEBUG [hbase-am-zkevent-worker-pool-2-thread-29] master.AssignmentManager(740): Handling transition=RS_ZK_REGION_OPENING, server=asf001.sp2.ygridcore.net,43802,1365336889418, region=d4e129e51e67d798e17083623fdde133, current state from region state map =null 2013-04-07 12:16:03,097 DEBUG [hbase-am-zkevent-worker-pool-2-thread-28] master.AssignmentManager(740): Handling transition=RS_ZK_REGION_OPENING, server=asf001.sp2.ygridcore.net,49518,1365336962323, region=682475e1358b7306bd3406e9330263a4, current state from region state map =null 2013-04-07 12:16:03,098 DEBUG [hbase-am-zkevent-worker-pool-2-thread-26] zookeeper.ZKUtil(1680): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Retrieved 156 byte(s) of data from znode /hbase/region-in-transition/dbb82f7f432d6664327da0b8f15b7d76 and set watcher; region=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,e,1365336962398.dbb82f7f432d6664327da0b8f15b7d76., state=RS_ZK_REGION_OPENING, servername=asf001.sp2.ygridcore.net,60884,1365336878389, createTime=1365336963043, payload.length=0 2013-04-07 12:16:03,101 DEBUG [hbase-am-zkevent-worker-pool-20-thread-29] zookeeper.ZKUtil(1680): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Retrieved 155 byte(s) of data from znode /hbase/region-in-transition/d4e129e51e67d798e17083623fdde133 and set watcher; region=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,,1365336962397.d4e129e51e67d798e17083623fdde133., state=RS_ZK_REGION_OPENING, servername=asf001.sp2.ygridcore.net,43802,1365336889418, createTime=1365336963049, payload.length=0 2013-04-07 12:16:03,101 DEBUG [hbase-am-zkevent-worker-pool-20-thread-29] master.AssignmentManager(740): Handling transition=RS_ZK_REGION_OPENING, server=asf001.sp2.ygridcore.net,43802,1365336889418, region=d4e129e51e67d798e17083623fdde133, current state from region state map ={testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,,1365336962397.d4e129e51e67d798e17083623fdde133. state=PENDING_OPEN, ts=1365336963016, server=asf001.sp2.ygridcore.net,43802,1365336889418} 2013-04-07 12:16:03,101 INFO [hbase-am-zkevent-worker-pool-20-thread-29] master.RegionStates(264): Region {NAME => 'testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,,1365336962397.d4e129e51e67d798e17083623fdde133.', STARTKEY => '', ENDKEY => 'a', ENCODED => d4e129e51e67d798e17083623fdde133,} transitioned from {testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,,1365336962397.d4e129e51e67d798e17083623fdde133. state=PENDING_OPEN, ts=1365336963016, server=asf001.sp2.ygridcore.net,43802,1365336889418} to {testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,,1365336962397.d4e129e51e67d798e17083623fdde133. state=OPENING, ts=1365336963101, server=asf001.sp2.ygridcore.net,43802,1365336889418} 2013-04-07 12:16:03,101 DEBUG [Thread-200-EventThread] zookeeper.ZooKeeperWatcher(274): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/region-in-transition/4432e800611c00a4ae168862a273fc70 2013-04-07 12:16:03,101 DEBUG [hbase-am-zkevent-worker-pool-20-thread-30] zookeeper.ZKUtil(1680): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Retrieved 156 byte(s) of data from znode /hbase/region-in-transition/eff8376859f5e6ba3201ec0ec9536d0f and set watcher; region=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,f,1365336962398.eff8376859f5e6ba3201ec0ec9536d0f., state=RS_ZK_REGION_OPENING, servername=asf001.sp2.ygridcore.net,60884,1365336878389, createTime=1365336963056, payload.length=0 2013-04-07 12:16:03,102 DEBUG [hbase-am-zkevent-worker-pool-20-thread-30] master.AssignmentManager(740): Handling transition=RS_ZK_REGION_OPENING, server=asf001.sp2.ygridcore.net,60884,1365336878389, region=eff8376859f5e6ba3201ec0ec9536d0f, current state from region state map ={testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,f,1365336962398.eff8376859f5e6ba3201ec0ec9536d0f. state=PENDING_OPEN, ts=1365336963018, server=asf001.sp2.ygridcore.net,60884,1365336878389} 2013-04-07 12:16:03,102 INFO [hbase-am-zkevent-worker-pool-20-thread-30] master.RegionStates(264): Region {NAME => 'testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,f,1365336962398.eff8376859f5e6ba3201ec0ec9536d0f.', STARTKEY => 'f', ENDKEY => 'g', ENCODED => eff8376859f5e6ba3201ec0ec9536d0f,} transitioned from {testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,f,1365336962398.eff8376859f5e6ba3201ec0ec9536d0f. state=PENDING_OPEN, ts=1365336963018, server=asf001.sp2.ygridcore.net,60884,1365336878389} to {testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,f,1365336962398.eff8376859f5e6ba3201ec0ec9536d0f. state=OPENING, ts=1365336963102, server=asf001.sp2.ygridcore.net,60884,1365336878389} 2013-04-07 12:16:03,102 DEBUG [RS_OPEN_REGION-asf001.sp2.ygridcore.net,60884,1365336878389-1] zookeeper.ZKAssign(784): regionserver:60884-0x13de46b5f8b0001 Successfully transitioned node eff8376859f5e6ba3201ec0ec9536d0f from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENING 2013-04-07 12:16:03,102 DEBUG [hbase-am-zkevent-worker-pool-2-thread-29] zookeeper.ZKUtil(1680): hconnection-0x1f327e-0x13de46b5f8b0003 Retrieved 43 byte(s) of data from znode /hbase/meta-region-server; data=asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:16:03,107 DEBUG [hbase-am-zkevent-worker-pool-2-thread-30] zookeeper.ZKUtil(1680): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Retrieved 156 byte(s) of data from znode /hbase/region-in-transition/eff8376859f5e6ba3201ec0ec9536d0f and set watcher; region=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,f,1365336962398.eff8376859f5e6ba3201ec0ec9536d0f., state=RS_ZK_REGION_OPENING, servername=asf001.sp2.ygridcore.net,60884,1365336878389, createTime=1365336963056, payload.length=0 2013-04-07 12:16:03,108 INFO [PostOpenDeployTasks:eff8376859f5e6ba3201ec0ec9536d0f] regionserver.HRegionServer(1563): Post open deploy tasks for region=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,f,1365336962398.eff8376859f5e6ba3201ec0ec9536d0f. 2013-04-07 12:16:03,109 DEBUG [hbase-am-zkevent-worker-pool-2-thread-29] zookeeper.ZKUtil(1680): hconnection-0x1f327e-0x13de46b5f8b0003 Retrieved 43 byte(s) of data from znode /hbase/meta-region-server; data=asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:16:03,111 DEBUG [hbase-am-zkevent-worker-pool-2-thread-31] zookeeper.ZKUtil(1680): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Retrieved 156 byte(s) of data from znode /hbase/region-in-transition/385e5846630b0cb4493758961a2c2457 and set watcher; region=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,i,1365336962398.385e5846630b0cb4493758961a2c2457., state=RS_ZK_REGION_OPENING, servername=asf001.sp2.ygridcore.net,60884,1365336878389, createTime=1365336963058, payload.length=0 2013-04-07 12:16:03,111 DEBUG [hbase-am-zkevent-worker-pool-20-thread-31] zookeeper.ZKUtil(1680): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Retrieved 156 byte(s) of data from znode /hbase/region-in-transition/385e5846630b0cb4493758961a2c2457 and set watcher; region=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,i,1365336962398.385e5846630b0cb4493758961a2c2457., state=RS_ZK_REGION_OPENING, servername=asf001.sp2.ygridcore.net,60884,1365336878389, createTime=1365336963058, payload.length=0 2013-04-07 12:16:03,111 DEBUG [hbase-am-zkevent-worker-pool-20-thread-31] master.AssignmentManager(740): Handling transition=RS_ZK_REGION_OPENING, server=asf001.sp2.ygridcore.net,60884,1365336878389, region=385e5846630b0cb4493758961a2c2457, current state from region state map ={testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,i,1365336962398.385e5846630b0cb4493758961a2c2457. state=PENDING_OPEN, ts=1365336963018, server=asf001.sp2.ygridcore.net,60884,1365336878389} 2013-04-07 12:16:03,112 INFO [hbase-am-zkevent-worker-pool-20-thread-31] master.RegionStates(264): Region {NAME => 'testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,i,1365336962398.385e5846630b0cb4493758961a2c2457.', STARTKEY => 'i', ENDKEY => 'j', ENCODED => 385e5846630b0cb4493758961a2c2457,} transitioned from {testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,i,1365336962398.385e5846630b0cb4493758961a2c2457. state=PENDING_OPEN, ts=1365336963018, server=asf001.sp2.ygridcore.net,60884,1365336878389} to {testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,i,1365336962398.385e5846630b0cb4493758961a2c2457. state=OPENING, ts=1365336963112, server=asf001.sp2.ygridcore.net,60884,1365336878389} 2013-04-07 12:16:03,112 DEBUG [hbase-am-zkevent-worker-pool-2-thread-32] zookeeper.ZKUtil(1680): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Retrieved 156 byte(s) of data from znode /hbase/region-in-transition/b1b340dff1e06fd6cb0b0a9c64964551 and set watcher; region=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,a,1365336962398.b1b340dff1e06fd6cb0b0a9c64964551., state=RS_ZK_REGION_OPENING, servername=asf001.sp2.ygridcore.net,49518,1365336962323, createTime=1365336963056, payload.length=0 2013-04-07 12:16:03,112 DEBUG [hbase-am-zkevent-worker-pool-20-thread-32] zookeeper.ZKUtil(1680): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Retrieved 156 byte(s) of data from znode /hbase/region-in-transition/b1b340dff1e06fd6cb0b0a9c64964551 and set watcher; region=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,a,1365336962398.b1b340dff1e06fd6cb0b0a9c64964551., state=RS_ZK_REGION_OPENING, servername=asf001.sp2.ygridcore.net,49518,1365336962323, createTime=1365336963056, payload.length=0 2013-04-07 12:16:03,112 DEBUG [hbase-am-zkevent-worker-pool-2-thread-23] zookeeper.ZKUtil(1680): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Retrieved 156 byte(s) of data from znode /hbase/region-in-transition/e99798d2c4d566ddb063efe03842457f and set watcher; region=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,d,1365336962398.e99798d2c4d566ddb063efe03842457f., state=RS_ZK_REGION_OPENING, servername=asf001.sp2.ygridcore.net,49518,1365336962323, createTime=1365336963053, payload.length=0 2013-04-07 12:16:03,112 DEBUG [hbase-am-zkevent-worker-pool-20-thread-32] master.AssignmentManager(740): Handling transition=RS_ZK_REGION_OPENING, server=asf001.sp2.ygridcore.net,49518,1365336962323, region=b1b340dff1e06fd6cb0b0a9c64964551, current state from region state map ={testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,a,1365336962398.b1b340dff1e06fd6cb0b0a9c64964551. state=PENDING_OPEN, ts=1365336963019, server=asf001.sp2.ygridcore.net,49518,1365336962323} 2013-04-07 12:16:03,112 INFO [hbase-am-zkevent-worker-pool-20-thread-32] master.RegionStates(264): Region {NAME => 'testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,a,1365336962398.b1b340dff1e06fd6cb0b0a9c64964551.', STARTKEY => 'a', ENDKEY => 'b', ENCODED => b1b340dff1e06fd6cb0b0a9c64964551,} transitioned from {testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,a,1365336962398.b1b340dff1e06fd6cb0b0a9c64964551. state=PENDING_OPEN, ts=1365336963019, server=asf001.sp2.ygridcore.net,49518,1365336962323} to {testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,a,1365336962398.b1b340dff1e06fd6cb0b0a9c64964551. state=OPENING, ts=1365336963112, server=asf001.sp2.ygridcore.net,49518,1365336962323} 2013-04-07 12:16:03,112 INFO [hbase-am-zkevent-worker-pool-2-thread-29] master.RegionStates(264): Region {NAME => 'testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,,1365336962397.d4e129e51e67d798e17083623fdde133.', STARTKEY => '', ENDKEY => 'a', ENCODED => d4e129e51e67d798e17083623fdde133,} transitioned from {testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,,1365336962397.d4e129e51e67d798e17083623fdde133. state=OFFLINE, ts=1365336963112, server=null} to {testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,,1365336962397.d4e129e51e67d798e17083623fdde133. state=OPENING, ts=1365336963112, server=asf001.sp2.ygridcore.net,43802,1365336889418} 2013-04-07 12:16:03,113 DEBUG [Thread-200-EventThread] zookeeper.ZooKeeperWatcher(274): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/region-in-transition/dbb82f7f432d6664327da0b8f15b7d76 2013-04-07 12:16:03,113 DEBUG [hbase-am-zkevent-worker-pool-2-thread-32] master.AssignmentManager(740): Handling transition=RS_ZK_REGION_OPENING, server=asf001.sp2.ygridcore.net,49518,1365336962323, region=b1b340dff1e06fd6cb0b0a9c64964551, current state from region state map =null 2013-04-07 12:16:03,113 DEBUG [hbase-am-zkevent-worker-pool-2-thread-22] zookeeper.ZKUtil(1680): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Retrieved 156 byte(s) of data from znode /hbase/region-in-transition/e99798d2c4d566ddb063efe03842457f and set watcher; region=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,d,1365336962398.e99798d2c4d566ddb063efe03842457f., state=RS_ZK_REGION_OPENING, servername=asf001.sp2.ygridcore.net,49518,1365336962323, createTime=1365336963053, payload.length=0 2013-04-07 12:16:03,113 DEBUG [hbase-am-zkevent-worker-pool-2-thread-26] master.AssignmentManager(740): Handling transition=RS_ZK_REGION_OPENING, server=asf001.sp2.ygridcore.net,60884,1365336878389, region=dbb82f7f432d6664327da0b8f15b7d76, current state from region state map =null 2013-04-07 12:16:03,113 DEBUG [hbase-am-zkevent-worker-pool-2-thread-30] master.AssignmentManager(740): Handling transition=RS_ZK_REGION_OPENING, server=asf001.sp2.ygridcore.net,60884,1365336878389, region=eff8376859f5e6ba3201ec0ec9536d0f, current state from region state map =null 2013-04-07 12:16:03,113 DEBUG [hbase-am-zkevent-worker-pool-2-thread-31] master.AssignmentManager(740): Handling transition=RS_ZK_REGION_OPENING, server=asf001.sp2.ygridcore.net,60884,1365336878389, region=385e5846630b0cb4493758961a2c2457, current state from region state map =null 2013-04-07 12:16:03,113 DEBUG [Thread-200-EventThread] zookeeper.ZooKeeperWatcher(274): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/region-in-transition/1ce6d62427641e2a0e135d3d27b54b5d 2013-04-07 12:16:03,115 DEBUG [hbase-am-zkevent-worker-pool-2-thread-28] zookeeper.ZKUtil(1680): hconnection-0x1f327e-0x13de46b5f8b0003 Retrieved 43 byte(s) of data from znode /hbase/meta-region-server; data=asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:16:03,116 DEBUG [Thread-200-EventThread] zookeeper.ZooKeeperWatcher(274): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/region-in-transition/d4e129e51e67d798e17083623fdde133 2013-04-07 12:16:03,117 DEBUG [hbase-am-zkevent-worker-pool-2-thread-28] zookeeper.ZKUtil(1680): hconnection-0x1f327e-0x13de46b5f8b0003 Retrieved 43 byte(s) of data from znode /hbase/meta-region-server; data=asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:16:03,117 DEBUG [PostOpenDeployTasks:eff8376859f5e6ba3201ec0ec9536d0f] zookeeper.ZKUtil(1680): hconnection-0x43ad4a-0x13de46b5f8b0005 Retrieved 43 byte(s) of data from znode /hbase/meta-region-server; data=asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:16:03,118 DEBUG [hbase-am-zkevent-worker-pool-2-thread-33] zookeeper.ZKUtil(1680): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Retrieved 156 byte(s) of data from znode /hbase/region-in-transition/4432e800611c00a4ae168862a273fc70 and set watcher; region=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,c,1365336962398.4432e800611c00a4ae168862a273fc70., state=RS_ZK_REGION_OPENING, servername=asf001.sp2.ygridcore.net,43802,1365336889418, createTime=1365336963071, payload.length=0 2013-04-07 12:16:03,119 INFO [RS_OPEN_REGION-asf001.sp2.ygridcore.net,60884,1365336878389-2] regionserver.HRegion(586): Onlined testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,i,1365336962398.385e5846630b0cb4493758961a2c2457.; next sequenceid=0 2013-04-07 12:16:03,119 INFO [hbase-am-zkevent-worker-pool-2-thread-28] master.RegionStates(264): Region {NAME => 'testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,g,1365336962398.682475e1358b7306bd3406e9330263a4.', STARTKEY => 'g', ENDKEY => 'h', ENCODED => 682475e1358b7306bd3406e9330263a4,} transitioned from {testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,g,1365336962398.682475e1358b7306bd3406e9330263a4. state=OFFLINE, ts=1365336963119, server=null} to {testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,g,1365336962398.682475e1358b7306bd3406e9330263a4. state=OPENING, ts=1365336963119, server=asf001.sp2.ygridcore.net,49518,1365336962323} 2013-04-07 12:16:03,119 DEBUG [RS_OPEN_REGION-asf001.sp2.ygridcore.net,60884,1365336878389-2] zookeeper.ZKAssign(721): regionserver:60884-0x13de46b5f8b0001 Attempting to transition node 385e5846630b0cb4493758961a2c2457 from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENING 2013-04-07 12:16:03,119 DEBUG [hbase-am-zkevent-worker-pool-2-thread-33] master.AssignmentManager(740): Handling transition=RS_ZK_REGION_OPENING, server=asf001.sp2.ygridcore.net,43802,1365336889418, region=4432e800611c00a4ae168862a273fc70, current state from region state map ={testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,c,1365336962398.4432e800611c00a4ae168862a273fc70. state=OPENING, ts=1365336963081, server=asf001.sp2.ygridcore.net,43802,1365336889418} 2013-04-07 12:16:03,119 DEBUG [PostOpenDeployTasks:eff8376859f5e6ba3201ec0ec9536d0f] zookeeper.ZKUtil(1680): hconnection-0x43ad4a-0x13de46b5f8b0005 Retrieved 43 byte(s) of data from znode /hbase/meta-region-server; data=asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:16:03,119 DEBUG [Thread-200-EventThread] zookeeper.ZooKeeperWatcher(274): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/region-in-transition/e99798d2c4d566ddb063efe03842457f 2013-04-07 12:16:03,120 DEBUG [hbase-am-zkevent-worker-pool-20-thread-33] zookeeper.ZKUtil(1680): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Retrieved 156 byte(s) of data from znode /hbase/region-in-transition/4432e800611c00a4ae168862a273fc70 and set watcher; region=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,c,1365336962398.4432e800611c00a4ae168862a273fc70., state=RS_ZK_REGION_OPENING, servername=asf001.sp2.ygridcore.net,43802,1365336889418, createTime=1365336963071, payload.length=0 2013-04-07 12:16:03,125 DEBUG [hbase-am-zkevent-worker-pool-20-thread-33] master.AssignmentManager(740): Handling transition=RS_ZK_REGION_OPENING, server=asf001.sp2.ygridcore.net,43802,1365336889418, region=4432e800611c00a4ae168862a273fc70, current state from region state map ={testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,c,1365336962398.4432e800611c00a4ae168862a273fc70. state=OPENING, ts=1365336963070, server=asf001.sp2.ygridcore.net,43802,1365336889418} 2013-04-07 12:16:03,125 INFO [hbase-am-zkevent-worker-pool-20-thread-33] master.RegionStates(264): Region {NAME => 'testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,c,1365336962398.4432e800611c00a4ae168862a273fc70.', STARTKEY => 'c', ENDKEY => 'd', ENCODED => 4432e800611c00a4ae168862a273fc70,} transitioned from {testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,c,1365336962398.4432e800611c00a4ae168862a273fc70. state=OPENING, ts=1365336963070, server=asf001.sp2.ygridcore.net,43802,1365336889418} to {testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,c,1365336962398.4432e800611c00a4ae168862a273fc70. state=OPENING, ts=1365336963125, server=asf001.sp2.ygridcore.net,43802,1365336889418} 2013-04-07 12:16:03,126 DEBUG [Thread-200-EventThread] zookeeper.ZooKeeperWatcher(274): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/region-in-transition/682475e1358b7306bd3406e9330263a4 2013-04-07 12:16:03,127 DEBUG [Thread-200-EventThread] zookeeper.ZooKeeperWatcher(274): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/region-in-transition/eff8376859f5e6ba3201ec0ec9536d0f 2013-04-07 12:16:03,132 DEBUG [RS_OPEN_REGION-asf001.sp2.ygridcore.net,60884,1365336878389-2] zookeeper.ZKUtil(1680): regionserver:60884-0x13de46b5f8b0001 Retrieved 156 byte(s) of data from znode /hbase/region-in-transition/385e5846630b0cb4493758961a2c2457; data=region=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,i,1365336962398.385e5846630b0cb4493758961a2c2457., state=RS_ZK_REGION_OPENING, servername=asf001.sp2.ygridcore.net,60884,1365336878389, createTime=1365336963058, payload.length=0 2013-04-07 12:16:03,132 DEBUG [hbase-am-zkevent-worker-pool-2-thread-31] zookeeper.ZKUtil(1680): hconnection-0x1f327e-0x13de46b5f8b0003 Retrieved 43 byte(s) of data from znode /hbase/meta-region-server; data=asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:16:03,134 DEBUG [hbase-am-zkevent-worker-pool-20-thread-34] zookeeper.ZKUtil(1680): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Retrieved 156 byte(s) of data from znode /hbase/region-in-transition/dbb82f7f432d6664327da0b8f15b7d76 and set watcher; region=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,e,1365336962398.dbb82f7f432d6664327da0b8f15b7d76., state=RS_ZK_REGION_OPENING, servername=asf001.sp2.ygridcore.net,60884,1365336878389, createTime=1365336963084, payload.length=0 2013-04-07 12:16:03,135 DEBUG [hbase-am-zkevent-worker-pool-20-thread-34] master.AssignmentManager(740): Handling transition=RS_ZK_REGION_OPENING, server=asf001.sp2.ygridcore.net,60884,1365336878389, region=dbb82f7f432d6664327da0b8f15b7d76, current state from region state map ={testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,e,1365336962398.dbb82f7f432d6664327da0b8f15b7d76. state=OPENING, ts=1365336963087, server=asf001.sp2.ygridcore.net,60884,1365336878389} 2013-04-07 12:16:03,136 INFO [hbase-am-zkevent-worker-pool-20-thread-34] master.RegionStates(264): Region {NAME => 'testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,e,1365336962398.dbb82f7f432d6664327da0b8f15b7d76.', STARTKEY => 'e', ENDKEY => 'f', ENCODED => dbb82f7f432d6664327da0b8f15b7d76,} transitioned from {testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,e,1365336962398.dbb82f7f432d6664327da0b8f15b7d76. state=OPENING, ts=1365336963087, server=asf001.sp2.ygridcore.net,60884,1365336878389} to {testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,e,1365336962398.dbb82f7f432d6664327da0b8f15b7d76. state=OPENING, ts=1365336963136, server=asf001.sp2.ygridcore.net,60884,1365336878389} 2013-04-07 12:16:03,138 DEBUG [hbase-am-zkevent-worker-pool-2-thread-34] zookeeper.ZKUtil(1680): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Retrieved 156 byte(s) of data from znode /hbase/region-in-transition/1ce6d62427641e2a0e135d3d27b54b5d and set watcher; region=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,h,1365336962398.1ce6d62427641e2a0e135d3d27b54b5d., state=RS_ZK_REGION_OPENING, servername=asf001.sp2.ygridcore.net,43802,1365336889418, createTime=1365336963086, payload.length=0 2013-04-07 12:16:03,141 DEBUG [Thread-200-EventThread] zookeeper.ZooKeeperWatcher(274): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/region-in-transition/385e5846630b0cb4493758961a2c2457 2013-04-07 12:16:03,141 DEBUG [RS_OPEN_REGION-asf001.sp2.ygridcore.net,60884,1365336878389-2] zookeeper.ZKAssign(784): regionserver:60884-0x13de46b5f8b0001 Successfully transitioned node 385e5846630b0cb4493758961a2c2457 from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENING 2013-04-07 12:16:03,141 INFO [PostOpenDeployTasks:385e5846630b0cb4493758961a2c2457] regionserver.HRegionServer(1563): Post open deploy tasks for region=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,i,1365336962398.385e5846630b0cb4493758961a2c2457. 2013-04-07 12:16:03,142 DEBUG [hbase-am-zkevent-worker-pool-2-thread-31] zookeeper.ZKUtil(1680): hconnection-0x1f327e-0x13de46b5f8b0003 Retrieved 43 byte(s) of data from znode /hbase/meta-region-server; data=asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:16:03,143 DEBUG [Thread-200-EventThread] zookeeper.ZooKeeperWatcher(274): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/region-in-transition/b1b340dff1e06fd6cb0b0a9c64964551 2013-04-07 12:16:03,144 DEBUG [Thread-200-EventThread] zookeeper.ZooKeeperWatcher(274): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/region-in-transition/4432e800611c00a4ae168862a273fc70 2013-04-07 12:16:03,144 INFO [hbase-am-zkevent-worker-pool-2-thread-31] master.RegionStates(264): Region {NAME => 'testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,i,1365336962398.385e5846630b0cb4493758961a2c2457.', STARTKEY => 'i', ENDKEY => 'j', ENCODED => 385e5846630b0cb4493758961a2c2457,} transitioned from {testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,i,1365336962398.385e5846630b0cb4493758961a2c2457. state=OFFLINE, ts=1365336963144, server=null} to {testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,i,1365336962398.385e5846630b0cb4493758961a2c2457. state=OPENING, ts=1365336963144, server=asf001.sp2.ygridcore.net,60884,1365336878389} 2013-04-07 12:16:03,145 DEBUG [hbase-am-zkevent-worker-pool-20-thread-35] zookeeper.ZKUtil(1680): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Retrieved 156 byte(s) of data from znode /hbase/region-in-transition/1ce6d62427641e2a0e135d3d27b54b5d and set watcher; region=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,h,1365336962398.1ce6d62427641e2a0e135d3d27b54b5d., state=RS_ZK_REGION_OPENING, servername=asf001.sp2.ygridcore.net,43802,1365336889418, createTime=1365336963086, payload.length=0 2013-04-07 12:16:03,145 DEBUG [hbase-am-zkevent-worker-pool-20-thread-35] master.AssignmentManager(740): Handling transition=RS_ZK_REGION_OPENING, server=asf001.sp2.ygridcore.net,43802,1365336889418, region=1ce6d62427641e2a0e135d3d27b54b5d, current state from region state map ={testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,h,1365336962398.1ce6d62427641e2a0e135d3d27b54b5d. state=OPENING, ts=1365336963073, server=asf001.sp2.ygridcore.net,43802,1365336889418} 2013-04-07 12:16:03,145 INFO [hbase-am-zkevent-worker-pool-20-thread-35] master.RegionStates(264): Region {NAME => 'testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,h,1365336962398.1ce6d62427641e2a0e135d3d27b54b5d.', STARTKEY => 'h', ENDKEY => 'i', ENCODED => 1ce6d62427641e2a0e135d3d27b54b5d,} transitioned from {testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,h,1365336962398.1ce6d62427641e2a0e135d3d27b54b5d. state=OPENING, ts=1365336963073, server=asf001.sp2.ygridcore.net,43802,1365336889418} to {testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,h,1365336962398.1ce6d62427641e2a0e135d3d27b54b5d. state=OPENING, ts=1365336963145, server=asf001.sp2.ygridcore.net,43802,1365336889418} 2013-04-07 12:16:03,146 DEBUG [hbase-am-zkevent-worker-pool-2-thread-30] zookeeper.ZKUtil(1680): hconnection-0x1f327e-0x13de46b5f8b0003 Retrieved 43 byte(s) of data from znode /hbase/meta-region-server; data=asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:16:03,148 DEBUG [hbase-am-zkevent-worker-pool-2-thread-30] zookeeper.ZKUtil(1680): hconnection-0x1f327e-0x13de46b5f8b0003 Retrieved 43 byte(s) of data from znode /hbase/meta-region-server; data=asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:16:03,149 INFO [hbase-am-zkevent-worker-pool-2-thread-30] master.RegionStates(264): Region {NAME => 'testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,f,1365336962398.eff8376859f5e6ba3201ec0ec9536d0f.', STARTKEY => 'f', ENDKEY => 'g', ENCODED => eff8376859f5e6ba3201ec0ec9536d0f,} transitioned from {testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,f,1365336962398.eff8376859f5e6ba3201ec0ec9536d0f. state=OFFLINE, ts=1365336963149, server=null} to {testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,f,1365336962398.eff8376859f5e6ba3201ec0ec9536d0f. state=OPENING, ts=1365336963149, server=asf001.sp2.ygridcore.net,60884,1365336878389} 2013-04-07 12:16:03,150 DEBUG [hbase-am-zkevent-worker-pool-2-thread-35] zookeeper.ZKUtil(1680): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Retrieved 155 byte(s) of data from znode /hbase/region-in-transition/d4e129e51e67d798e17083623fdde133 and set watcher; region=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,,1365336962397.d4e129e51e67d798e17083623fdde133., state=RS_ZK_REGION_OPENED, servername=asf001.sp2.ygridcore.net,43802,1365336889418, createTime=1365336963136, payload.length=0 2013-04-07 12:16:03,151 DEBUG [hbase-am-zkevent-worker-pool-20-thread-36] zookeeper.ZKUtil(1680): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Retrieved 155 byte(s) of data from znode /hbase/region-in-transition/d4e129e51e67d798e17083623fdde133 and set watcher; region=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,,1365336962397.d4e129e51e67d798e17083623fdde133., state=RS_ZK_REGION_OPENED, servername=asf001.sp2.ygridcore.net,43802,1365336889418, createTime=1365336963136, payload.length=0 2013-04-07 12:16:03,151 DEBUG [hbase-am-zkevent-worker-pool-20-thread-37] zookeeper.ZKUtil(1680): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Retrieved 156 byte(s) of data from znode /hbase/region-in-transition/e99798d2c4d566ddb063efe03842457f and set watcher; region=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,d,1365336962398.e99798d2c4d566ddb063efe03842457f., state=RS_ZK_REGION_OPENING, servername=asf001.sp2.ygridcore.net,49518,1365336962323, createTime=1365336963094, payload.length=0 2013-04-07 12:16:03,151 DEBUG [hbase-am-zkevent-worker-pool-20-thread-36] master.AssignmentManager(740): Handling transition=RS_ZK_REGION_OPENED, server=asf001.sp2.ygridcore.net,43802,1365336889418, region=d4e129e51e67d798e17083623fdde133, current state from region state map ={testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,,1365336962397.d4e129e51e67d798e17083623fdde133. state=OPENING, ts=1365336963101, server=asf001.sp2.ygridcore.net,43802,1365336889418} 2013-04-07 12:16:03,151 DEBUG [hbase-am-zkevent-worker-pool-20-thread-37] master.AssignmentManager(740): Handling transition=RS_ZK_REGION_OPENING, server=asf001.sp2.ygridcore.net,49518,1365336962323, region=e99798d2c4d566ddb063efe03842457f, current state from region state map ={testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,d,1365336962398.e99798d2c4d566ddb063efe03842457f. state=OPENING, ts=1365336963078, server=asf001.sp2.ygridcore.net,49518,1365336962323} 2013-04-07 12:16:03,151 INFO [hbase-am-zkevent-worker-pool-20-thread-36] master.RegionStates(264): Region {NAME => 'testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,,1365336962397.d4e129e51e67d798e17083623fdde133.', STARTKEY => '', ENDKEY => 'a', ENCODED => d4e129e51e67d798e17083623fdde133,} transitioned from {testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,,1365336962397.d4e129e51e67d798e17083623fdde133. state=OPENING, ts=1365336963101, server=asf001.sp2.ygridcore.net,43802,1365336889418} to {testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,,1365336962397.d4e129e51e67d798e17083623fdde133. state=OPEN, ts=1365336963151, server=asf001.sp2.ygridcore.net,43802,1365336889418} 2013-04-07 12:16:03,151 DEBUG [hbase-am-zkevent-worker-pool-2-thread-37] zookeeper.ZKUtil(1680): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Retrieved 156 byte(s) of data from znode /hbase/region-in-transition/682475e1358b7306bd3406e9330263a4 and set watcher; region=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,g,1365336962398.682475e1358b7306bd3406e9330263a4., state=RS_ZK_REGION_OPENING, servername=asf001.sp2.ygridcore.net,49518,1365336962323, createTime=1365336963096, payload.length=0 2013-04-07 12:16:03,151 INFO [hbase-am-zkevent-worker-pool-20-thread-37] master.RegionStates(264): Region {NAME => 'testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,d,1365336962398.e99798d2c4d566ddb063efe03842457f.', STARTKEY => 'd', ENDKEY => 'e', ENCODED => e99798d2c4d566ddb063efe03842457f,} transitioned from {testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,d,1365336962398.e99798d2c4d566ddb063efe03842457f. state=OPENING, ts=1365336963078, server=asf001.sp2.ygridcore.net,49518,1365336962323} to {testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,d,1365336962398.e99798d2c4d566ddb063efe03842457f. state=OPENING, ts=1365336963151, server=asf001.sp2.ygridcore.net,49518,1365336962323} 2013-04-07 12:16:03,151 DEBUG [hbase-am-zkevent-worker-pool-20-thread-39] zookeeper.ZKUtil(1680): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Retrieved 156 byte(s) of data from znode /hbase/region-in-transition/eff8376859f5e6ba3201ec0ec9536d0f and set watcher; region=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,f,1365336962398.eff8376859f5e6ba3201ec0ec9536d0f., state=RS_ZK_REGION_OPENING, servername=asf001.sp2.ygridcore.net,60884,1365336878389, createTime=1365336963094, payload.length=0 2013-04-07 12:16:03,152 DEBUG [MASTER_OPEN_REGION-asf001.sp2.ygridcore.net,33796,1365336878036-2] handler.OpenedRegionHandler(146): Handling OPENED event for testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,,1365336962397.d4e129e51e67d798e17083623fdde133. from asf001.sp2.ygridcore.net,43802,1365336889418; deleting unassigned node 2013-04-07 12:16:03,152 DEBUG [hbase-am-zkevent-worker-pool-20-thread-39] master.AssignmentManager(740): Handling transition=RS_ZK_REGION_OPENING, server=asf001.sp2.ygridcore.net,60884,1365336878389, region=eff8376859f5e6ba3201ec0ec9536d0f, current state from region state map ={testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,f,1365336962398.eff8376859f5e6ba3201ec0ec9536d0f. state=OPENING, ts=1365336963102, server=asf001.sp2.ygridcore.net,60884,1365336878389} 2013-04-07 12:16:03,152 DEBUG [MASTER_OPEN_REGION-asf001.sp2.ygridcore.net,33796,1365336878036-2] zookeeper.ZKAssign(405): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Deleting existing unassigned node for d4e129e51e67d798e17083623fdde133 that is in expected state RS_ZK_REGION_OPENED 2013-04-07 12:16:03,152 INFO [hbase-am-zkevent-worker-pool-20-thread-39] master.RegionStates(264): Region {NAME => 'testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,f,1365336962398.eff8376859f5e6ba3201ec0ec9536d0f.', STARTKEY => 'f', ENDKEY => 'g', ENCODED => eff8376859f5e6ba3201ec0ec9536d0f,} transitioned from {testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,f,1365336962398.eff8376859f5e6ba3201ec0ec9536d0f. state=OPENING, ts=1365336963102, server=asf001.sp2.ygridcore.net,60884,1365336878389} to {testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,f,1365336962398.eff8376859f5e6ba3201ec0ec9536d0f. state=OPENING, ts=1365336963152, server=asf001.sp2.ygridcore.net,60884,1365336878389} 2013-04-07 12:16:03,152 DEBUG [MASTER_TABLE_OPERATIONS-asf001.sp2.ygridcore.net,33796,1365336878036-0] lock.ZKInterProcessLockBase(297): Successfully released /hbase/table-lock/testRegionAssignmentAfterMasterRecoveryDueToZKExpiry/write-master:337960000000000 2013-04-07 12:16:03,152 DEBUG [MASTER_TABLE_OPERATIONS-asf001.sp2.ygridcore.net,33796,1365336878036-0] master.TableLockManager$ZKTableLockManager$TableLockImpl(269): Released table lock on :testRegionAssignmentAfterMasterRecoveryDueToZKExpiry 2013-04-07 12:16:03,152 DEBUG [hbase-am-zkevent-worker-pool-2-thread-36] zookeeper.ZKUtil(1680): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Retrieved 156 byte(s) of data from znode /hbase/region-in-transition/e99798d2c4d566ddb063efe03842457f and set watcher; region=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,d,1365336962398.e99798d2c4d566ddb063efe03842457f., state=RS_ZK_REGION_OPENING, servername=asf001.sp2.ygridcore.net,49518,1365336962323, createTime=1365336963094, payload.length=0 2013-04-07 12:16:03,152 DEBUG [hbase-am-zkevent-worker-pool-20-thread-38] zookeeper.ZKUtil(1680): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Retrieved 156 byte(s) of data from znode /hbase/region-in-transition/682475e1358b7306bd3406e9330263a4 and set watcher; region=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,g,1365336962398.682475e1358b7306bd3406e9330263a4., state=RS_ZK_REGION_OPENING, servername=asf001.sp2.ygridcore.net,49518,1365336962323, createTime=1365336963096, payload.length=0 2013-04-07 12:16:03,153 DEBUG [hbase-am-zkevent-worker-pool-20-thread-38] master.AssignmentManager(740): Handling transition=RS_ZK_REGION_OPENING, server=asf001.sp2.ygridcore.net,49518,1365336962323, region=682475e1358b7306bd3406e9330263a4, current state from region state map ={testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,g,1365336962398.682475e1358b7306bd3406e9330263a4. state=OPENING, ts=1365336963082, server=asf001.sp2.ygridcore.net,49518,1365336962323} 2013-04-07 12:16:03,153 DEBUG [hbase-am-zkevent-worker-pool-20-thread-21] zookeeper.ZKUtil(1680): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Retrieved 156 byte(s) of data from znode /hbase/region-in-transition/b1b340dff1e06fd6cb0b0a9c64964551 and set watcher; region=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,a,1365336962398.b1b340dff1e06fd6cb0b0a9c64964551., state=RS_ZK_REGION_OPENING, servername=asf001.sp2.ygridcore.net,49518,1365336962323, createTime=1365336963133, payload.length=0 2013-04-07 12:16:03,153 INFO [hbase-am-zkevent-worker-pool-20-thread-38] master.RegionStates(264): Region {NAME => 'testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,g,1365336962398.682475e1358b7306bd3406e9330263a4.', STARTKEY => 'g', ENDKEY => 'h', ENCODED => 682475e1358b7306bd3406e9330263a4,} transitioned from {testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,g,1365336962398.682475e1358b7306bd3406e9330263a4. state=OPENING, ts=1365336963082, server=asf001.sp2.ygridcore.net,49518,1365336962323} to {testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,g,1365336962398.682475e1358b7306bd3406e9330263a4. state=OPENING, ts=1365336963153, server=asf001.sp2.ygridcore.net,49518,1365336962323} 2013-04-07 12:16:03,153 DEBUG [hbase-am-zkevent-worker-pool-20-thread-21] master.AssignmentManager(740): Handling transition=RS_ZK_REGION_OPENING, server=asf001.sp2.ygridcore.net,49518,1365336962323, region=b1b340dff1e06fd6cb0b0a9c64964551, current state from region state map ={testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,a,1365336962398.b1b340dff1e06fd6cb0b0a9c64964551. state=OPENING, ts=1365336963112, server=asf001.sp2.ygridcore.net,49518,1365336962323} 2013-04-07 12:16:03,153 DEBUG [hbase-am-zkevent-worker-pool-20-thread-40] zookeeper.ZKUtil(1680): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Retrieved 156 byte(s) of data from znode /hbase/region-in-transition/385e5846630b0cb4493758961a2c2457 and set watcher; region=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,i,1365336962398.385e5846630b0cb4493758961a2c2457., state=RS_ZK_REGION_OPENING, servername=asf001.sp2.ygridcore.net,60884,1365336878389, createTime=1365336963132, payload.length=0 2013-04-07 12:16:03,153 INFO [hbase-am-zkevent-worker-pool-20-thread-21] master.RegionStates(264): Region {NAME => 'testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,a,1365336962398.b1b340dff1e06fd6cb0b0a9c64964551.', STARTKEY => 'a', ENDKEY => 'b', ENCODED => b1b340dff1e06fd6cb0b0a9c64964551,} transitioned from {testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,a,1365336962398.b1b340dff1e06fd6cb0b0a9c64964551. state=OPENING, ts=1365336963112, server=asf001.sp2.ygridcore.net,49518,1365336962323} to {testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,a,1365336962398.b1b340dff1e06fd6cb0b0a9c64964551. state=OPENING, ts=1365336963153, server=asf001.sp2.ygridcore.net,49518,1365336962323} 2013-04-07 12:16:03,153 DEBUG [hbase-am-zkevent-worker-pool-20-thread-40] master.AssignmentManager(740): Handling transition=RS_ZK_REGION_OPENING, server=asf001.sp2.ygridcore.net,60884,1365336878389, region=385e5846630b0cb4493758961a2c2457, current state from region state map ={testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,i,1365336962398.385e5846630b0cb4493758961a2c2457. state=OPENING, ts=1365336963112, server=asf001.sp2.ygridcore.net,60884,1365336878389} 2013-04-07 12:16:03,154 INFO [hbase-am-zkevent-worker-pool-20-thread-40] master.RegionStates(264): Region {NAME => 'testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,i,1365336962398.385e5846630b0cb4493758961a2c2457.', STARTKEY => 'i', ENDKEY => 'j', ENCODED => 385e5846630b0cb4493758961a2c2457,} transitioned from {testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,i,1365336962398.385e5846630b0cb4493758961a2c2457. state=OPENING, ts=1365336963112, server=asf001.sp2.ygridcore.net,60884,1365336878389} to {testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,i,1365336962398.385e5846630b0cb4493758961a2c2457. state=OPENING, ts=1365336963154, server=asf001.sp2.ygridcore.net,60884,1365336878389} 2013-04-07 12:16:03,156 DEBUG [hbase-am-zkevent-worker-pool-20-thread-22] zookeeper.ZKUtil(1680): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Retrieved 156 byte(s) of data from znode /hbase/region-in-transition/4432e800611c00a4ae168862a273fc70 and set watcher; region=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,c,1365336962398.4432e800611c00a4ae168862a273fc70., state=RS_ZK_REGION_OPENED, servername=asf001.sp2.ygridcore.net,43802,1365336889418, createTime=1365336963133, payload.length=0 2013-04-07 12:16:03,156 DEBUG [hbase-am-zkevent-worker-pool-20-thread-22] master.AssignmentManager(740): Handling transition=RS_ZK_REGION_OPENED, server=asf001.sp2.ygridcore.net,43802,1365336889418, region=4432e800611c00a4ae168862a273fc70, current state from region state map ={testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,c,1365336962398.4432e800611c00a4ae168862a273fc70. state=OPENING, ts=1365336963125, server=asf001.sp2.ygridcore.net,43802,1365336889418} 2013-04-07 12:16:03,156 INFO [hbase-am-zkevent-worker-pool-20-thread-22] master.RegionStates(264): Region {NAME => 'testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,c,1365336962398.4432e800611c00a4ae168862a273fc70.', STARTKEY => 'c', ENDKEY => 'd', ENCODED => 4432e800611c00a4ae168862a273fc70,} transitioned from {testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,c,1365336962398.4432e800611c00a4ae168862a273fc70. state=OPENING, ts=1365336963125, server=asf001.sp2.ygridcore.net,43802,1365336889418} to {testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,c,1365336962398.4432e800611c00a4ae168862a273fc70. state=OPEN, ts=1365336963156, server=asf001.sp2.ygridcore.net,43802,1365336889418} 2013-04-07 12:16:03,157 DEBUG [MASTER_OPEN_REGION-asf001.sp2.ygridcore.net,33796,1365336878036-3] handler.OpenedRegionHandler(146): Handling OPENED event for testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,c,1365336962398.4432e800611c00a4ae168862a273fc70. from asf001.sp2.ygridcore.net,43802,1365336889418; deleting unassigned node 2013-04-07 12:16:03,157 DEBUG [MASTER_OPEN_REGION-asf001.sp2.ygridcore.net,33796,1365336878036-3] zookeeper.ZKAssign(405): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Deleting existing unassigned node for 4432e800611c00a4ae168862a273fc70 that is in expected state RS_ZK_REGION_OPENED 2013-04-07 12:16:03,157 DEBUG [hbase-am-zkevent-worker-pool-2-thread-26] zookeeper.ZKUtil(1680): hconnection-0x1f327e-0x13de46b5f8b0003 Retrieved 43 byte(s) of data from znode /hbase/meta-region-server; data=asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:16:03,158 DEBUG [Thread-200-EventThread] zookeeper.ZooKeeperWatcher(274): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/region-in-transition/eff8376859f5e6ba3201ec0ec9536d0f 2013-04-07 12:16:03,159 DEBUG [PostOpenDeployTasks:385e5846630b0cb4493758961a2c2457] zookeeper.ZKUtil(1680): hconnection-0x43ad4a-0x13de46b5f8b0005 Retrieved 43 byte(s) of data from znode /hbase/meta-region-server; data=asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:16:03,159 DEBUG [Thread-200-EventThread] zookeeper.ZooKeeperWatcher(274): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/region-in-transition/dbb82f7f432d6664327da0b8f15b7d76 2013-04-07 12:16:03,163 DEBUG [hbase-am-zkevent-worker-pool-2-thread-38] zookeeper.ZKUtil(1680): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Retrieved 156 byte(s) of data from znode /hbase/region-in-transition/385e5846630b0cb4493758961a2c2457 and set watcher; region=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,i,1365336962398.385e5846630b0cb4493758961a2c2457., state=RS_ZK_REGION_OPENING, servername=asf001.sp2.ygridcore.net,60884,1365336878389, createTime=1365336963132, payload.length=0 2013-04-07 12:16:03,164 INFO [PostOpenDeployTasks:385e5846630b0cb4493758961a2c2457] catalog.MetaEditor(432): Updated row testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,i,1365336962398.385e5846630b0cb4493758961a2c2457. with server=asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:16:03,164 INFO [PostOpenDeployTasks:385e5846630b0cb4493758961a2c2457] regionserver.HRegionServer(1584): Done with post open deploy task for region=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,i,1365336962398.385e5846630b0cb4493758961a2c2457. 2013-04-07 12:16:03,165 DEBUG [RS_OPEN_REGION-asf001.sp2.ygridcore.net,60884,1365336878389-2] zookeeper.ZKAssign(721): regionserver:60884-0x13de46b5f8b0001 Attempting to transition node 385e5846630b0cb4493758961a2c2457 from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENED 2013-04-07 12:16:03,168 DEBUG [hbase-am-zkevent-worker-pool-2-thread-26] zookeeper.ZKUtil(1680): hconnection-0x1f327e-0x13de46b5f8b0003 Retrieved 43 byte(s) of data from znode /hbase/meta-region-server; data=asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:16:03,171 INFO [hbase-am-zkevent-worker-pool-2-thread-26] master.RegionStates(264): Region {NAME => 'testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,e,1365336962398.dbb82f7f432d6664327da0b8f15b7d76.', STARTKEY => 'e', ENDKEY => 'f', ENCODED => dbb82f7f432d6664327da0b8f15b7d76,} transitioned from {testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,e,1365336962398.dbb82f7f432d6664327da0b8f15b7d76. state=OFFLINE, ts=1365336963171, server=null} to {testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,e,1365336962398.dbb82f7f432d6664327da0b8f15b7d76. state=OPENING, ts=1365336963171, server=asf001.sp2.ygridcore.net,60884,1365336878389} 2013-04-07 12:16:03,176 DEBUG [hbase-am-zkevent-worker-pool-2-thread-39] zookeeper.ZKUtil(1680): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Retrieved 156 byte(s) of data from znode /hbase/region-in-transition/eff8376859f5e6ba3201ec0ec9536d0f and set watcher; region=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,f,1365336962398.eff8376859f5e6ba3201ec0ec9536d0f., state=RS_ZK_REGION_OPENED, servername=asf001.sp2.ygridcore.net,60884,1365336878389, createTime=1365336963154, payload.length=0 2013-04-07 12:16:03,176 DEBUG [RS_OPEN_REGION-asf001.sp2.ygridcore.net,60884,1365336878389-1] regionserver.HRegion(496): Instantiated testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,b,1365336962398.11c8123e8d563f264d947ffa106e4c7c. 2013-04-07 12:16:03,176 DEBUG [hbase-am-zkevent-worker-pool-2-thread-32] zookeeper.ZKUtil(1680): hconnection-0x1f327e-0x13de46b5f8b0003 Retrieved 43 byte(s) of data from znode /hbase/meta-region-server; data=asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:16:03,178 DEBUG [MASTER_OPEN_REGION-asf001.sp2.ygridcore.net,33796,1365336878036-3] zookeeper.ZKUtil(1680): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Retrieved 156 byte(s) of data from znode /hbase/region-in-transition/4432e800611c00a4ae168862a273fc70; data=region=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,c,1365336962398.4432e800611c00a4ae168862a273fc70., state=RS_ZK_REGION_OPENED, servername=asf001.sp2.ygridcore.net,43802,1365336889418, createTime=1365336963133, payload.length=0 2013-04-07 12:16:03,178 DEBUG [hbase-am-zkevent-worker-pool-2-thread-32] zookeeper.ZKUtil(1680): hconnection-0x1f327e-0x13de46b5f8b0003 Retrieved 43 byte(s) of data from znode /hbase/meta-region-server; data=asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:16:03,178 DEBUG [hbase-am-zkevent-worker-pool-20-thread-23] zookeeper.ZKUtil(1680): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Retrieved 156 byte(s) of data from znode /hbase/region-in-transition/eff8376859f5e6ba3201ec0ec9536d0f and set watcher; region=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,f,1365336962398.eff8376859f5e6ba3201ec0ec9536d0f., state=RS_ZK_REGION_OPENED, servername=asf001.sp2.ygridcore.net,60884,1365336878389, createTime=1365336963154, payload.length=0 2013-04-07 12:16:03,178 DEBUG [hbase-am-zkevent-worker-pool-20-thread-23] master.AssignmentManager(740): Handling transition=RS_ZK_REGION_OPENED, server=asf001.sp2.ygridcore.net,60884,1365336878389, region=eff8376859f5e6ba3201ec0ec9536d0f, current state from region state map ={testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,f,1365336962398.eff8376859f5e6ba3201ec0ec9536d0f. state=OPENING, ts=1365336963152, server=asf001.sp2.ygridcore.net,60884,1365336878389} 2013-04-07 12:16:03,179 INFO [hbase-am-zkevent-worker-pool-20-thread-23] master.RegionStates(264): Region {NAME => 'testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,f,1365336962398.eff8376859f5e6ba3201ec0ec9536d0f.', STARTKEY => 'f', ENDKEY => 'g', ENCODED => eff8376859f5e6ba3201ec0ec9536d0f,} transitioned from {testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,f,1365336962398.eff8376859f5e6ba3201ec0ec9536d0f. state=OPENING, ts=1365336963152, server=asf001.sp2.ygridcore.net,60884,1365336878389} to {testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,f,1365336962398.eff8376859f5e6ba3201ec0ec9536d0f. state=OPEN, ts=1365336963179, server=asf001.sp2.ygridcore.net,60884,1365336878389} 2013-04-07 12:16:03,179 DEBUG [MASTER_OPEN_REGION-asf001.sp2.ygridcore.net,33796,1365336878036-1] handler.OpenedRegionHandler(146): Handling OPENED event for testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,f,1365336962398.eff8376859f5e6ba3201ec0ec9536d0f. from asf001.sp2.ygridcore.net,60884,1365336878389; deleting unassigned node 2013-04-07 12:16:03,179 DEBUG [MASTER_OPEN_REGION-asf001.sp2.ygridcore.net,33796,1365336878036-1] zookeeper.ZKAssign(405): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Deleting existing unassigned node for eff8376859f5e6ba3201ec0ec9536d0f that is in expected state RS_ZK_REGION_OPENED 2013-04-07 12:16:03,179 DEBUG [Thread-200-EventThread] zookeeper.ZooKeeperWatcher(274): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/region-in-transition/682475e1358b7306bd3406e9330263a4 2013-04-07 12:16:03,180 INFO [hbase-am-zkevent-worker-pool-2-thread-32] master.RegionStates(264): Region {NAME => 'testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,a,1365336962398.b1b340dff1e06fd6cb0b0a9c64964551.', STARTKEY => 'a', ENDKEY => 'b', ENCODED => b1b340dff1e06fd6cb0b0a9c64964551,} transitioned from {testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,a,1365336962398.b1b340dff1e06fd6cb0b0a9c64964551. state=OFFLINE, ts=1365336963180, server=null} to {testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,a,1365336962398.b1b340dff1e06fd6cb0b0a9c64964551. state=OPENING, ts=1365336963180, server=asf001.sp2.ygridcore.net,49518,1365336962323} 2013-04-07 12:16:03,180 DEBUG [hbase-am-zkevent-worker-pool-2-thread-38] master.AssignmentManager(740): Handling transition=RS_ZK_REGION_OPENING, server=asf001.sp2.ygridcore.net,60884,1365336878389, region=385e5846630b0cb4493758961a2c2457, current state from region state map ={testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,i,1365336962398.385e5846630b0cb4493758961a2c2457. state=OPENING, ts=1365336963144, server=asf001.sp2.ygridcore.net,60884,1365336878389} 2013-04-07 12:16:03,180 DEBUG [hbase-am-zkevent-worker-pool-2-thread-36] master.AssignmentManager(740): Handling transition=RS_ZK_REGION_OPENING, server=asf001.sp2.ygridcore.net,49518,1365336962323, region=e99798d2c4d566ddb063efe03842457f, current state from region state map ={testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,d,1365336962398.e99798d2c4d566ddb063efe03842457f. state=OPENING, ts=1365336963097, server=asf001.sp2.ygridcore.net,49518,1365336962323} 2013-04-07 12:16:03,180 DEBUG [hbase-am-zkevent-worker-pool-20-thread-24] zookeeper.ZKUtil(1680): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Retrieved 156 byte(s) of data from znode /hbase/region-in-transition/dbb82f7f432d6664327da0b8f15b7d76 and set watcher; region=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,e,1365336962398.dbb82f7f432d6664327da0b8f15b7d76., state=RS_ZK_REGION_OPENED, servername=asf001.sp2.ygridcore.net,60884,1365336878389, createTime=1365336963155, payload.length=0 2013-04-07 12:16:03,181 DEBUG [hbase-am-zkevent-worker-pool-20-thread-24] master.AssignmentManager(740): Handling transition=RS_ZK_REGION_OPENED, server=asf001.sp2.ygridcore.net,60884,1365336878389, region=dbb82f7f432d6664327da0b8f15b7d76, current state from region state map ={testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,e,1365336962398.dbb82f7f432d6664327da0b8f15b7d76. state=OPENING, ts=1365336963136, server=asf001.sp2.ygridcore.net,60884,1365336878389} 2013-04-07 12:16:03,181 INFO [hbase-am-zkevent-worker-pool-20-thread-24] master.RegionStates(264): Region {NAME => 'testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,e,1365336962398.dbb82f7f432d6664327da0b8f15b7d76.', STARTKEY => 'e', ENDKEY => 'f', ENCODED => dbb82f7f432d6664327da0b8f15b7d76,} transitioned from {testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,e,1365336962398.dbb82f7f432d6664327da0b8f15b7d76. state=OPENING, ts=1365336963136, server=asf001.sp2.ygridcore.net,60884,1365336878389} to {testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,e,1365336962398.dbb82f7f432d6664327da0b8f15b7d76. state=OPEN, ts=1365336963181, server=asf001.sp2.ygridcore.net,60884,1365336878389} 2013-04-07 12:16:03,182 DEBUG [MASTER_OPEN_REGION-asf001.sp2.ygridcore.net,33796,1365336878036-4] handler.OpenedRegionHandler(146): Handling OPENED event for testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,e,1365336962398.dbb82f7f432d6664327da0b8f15b7d76. from asf001.sp2.ygridcore.net,60884,1365336878389; deleting unassigned node 2013-04-07 12:16:03,182 DEBUG [MASTER_OPEN_REGION-asf001.sp2.ygridcore.net,33796,1365336878036-4] zookeeper.ZKAssign(405): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Deleting existing unassigned node for dbb82f7f432d6664327da0b8f15b7d76 that is in expected state RS_ZK_REGION_OPENED 2013-04-07 12:16:03,182 INFO [RS_OPEN_REGION-asf001.sp2.ygridcore.net,60884,1365336878389-1] regionserver.HRegion(586): Onlined testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,b,1365336962398.11c8123e8d563f264d947ffa106e4c7c.; next sequenceid=0 2013-04-07 12:16:03,182 DEBUG [RS_OPEN_REGION-asf001.sp2.ygridcore.net,60884,1365336878389-1] zookeeper.ZKAssign(721): regionserver:60884-0x13de46b5f8b0001 Attempting to transition node 11c8123e8d563f264d947ffa106e4c7c from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENING 2013-04-07 12:16:03,181 INFO [hbase-am-zkevent-worker-pool-2-thread-38] master.RegionStates(264): Region {NAME => 'testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,i,1365336962398.385e5846630b0cb4493758961a2c2457.', STARTKEY => 'i', ENDKEY => 'j', ENCODED => 385e5846630b0cb4493758961a2c2457,} transitioned from {testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,i,1365336962398.385e5846630b0cb4493758961a2c2457. state=OPENING, ts=1365336963144, server=asf001.sp2.ygridcore.net,60884,1365336878389} to {testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,i,1365336962398.385e5846630b0cb4493758961a2c2457. state=OPENING, ts=1365336963181, server=asf001.sp2.ygridcore.net,60884,1365336878389} 2013-04-07 12:16:03,182 DEBUG [Thread-200-EventThread] zookeeper.ZooKeeperWatcher(274): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/region-in-transition/e99798d2c4d566ddb063efe03842457f 2013-04-07 12:16:03,183 DEBUG [hbase-am-zkevent-worker-pool-2-thread-37] master.AssignmentManager(740): Handling transition=RS_ZK_REGION_OPENING, server=asf001.sp2.ygridcore.net,49518,1365336962323, region=682475e1358b7306bd3406e9330263a4, current state from region state map ={testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,g,1365336962398.682475e1358b7306bd3406e9330263a4. state=OPENING, ts=1365336963119, server=asf001.sp2.ygridcore.net,49518,1365336962323} 2013-04-07 12:16:03,183 DEBUG [hbase-am-zkevent-worker-pool-2-thread-34] master.AssignmentManager(740): Handling transition=RS_ZK_REGION_OPENING, server=asf001.sp2.ygridcore.net,43802,1365336889418, region=1ce6d62427641e2a0e135d3d27b54b5d, current state from region state map ={testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,h,1365336962398.1ce6d62427641e2a0e135d3d27b54b5d. state=OPENING, ts=1365336963089, server=asf001.sp2.ygridcore.net,43802,1365336889418} 2013-04-07 12:16:03,183 DEBUG [hbase-am-zkevent-worker-pool-2-thread-35] master.AssignmentManager(740): Handling transition=RS_ZK_REGION_OPENED, server=asf001.sp2.ygridcore.net,43802,1365336889418, region=d4e129e51e67d798e17083623fdde133, current state from region state map ={testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,,1365336962397.d4e129e51e67d798e17083623fdde133. state=OPENING, ts=1365336963112, server=asf001.sp2.ygridcore.net,43802,1365336889418} 2013-04-07 12:16:03,183 INFO [hbase-am-zkevent-worker-pool-2-thread-33] master.RegionStates(264): Region {NAME => 'testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,c,1365336962398.4432e800611c00a4ae168862a273fc70.', STARTKEY => 'c', ENDKEY => 'd', ENCODED => 4432e800611c00a4ae168862a273fc70,} transitioned from {testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,c,1365336962398.4432e800611c00a4ae168862a273fc70. state=OPENING, ts=1365336963081, server=asf001.sp2.ygridcore.net,43802,1365336889418} to {testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,c,1365336962398.4432e800611c00a4ae168862a273fc70. state=OPENING, ts=1365336963183, server=asf001.sp2.ygridcore.net,43802,1365336889418} 2013-04-07 12:16:03,183 DEBUG [hbase-am-zkevent-worker-pool-2-thread-39] master.AssignmentManager(740): Handling transition=RS_ZK_REGION_OPENED, server=asf001.sp2.ygridcore.net,60884,1365336878389, region=eff8376859f5e6ba3201ec0ec9536d0f, current state from region state map ={testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,f,1365336962398.eff8376859f5e6ba3201ec0ec9536d0f. state=OPENING, ts=1365336963149, server=asf001.sp2.ygridcore.net,60884,1365336878389} 2013-04-07 12:16:03,184 INFO [hbase-am-zkevent-worker-pool-2-thread-36] master.RegionStates(264): Region {NAME => 'testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,d,1365336962398.e99798d2c4d566ddb063efe03842457f.', STARTKEY => 'd', ENDKEY => 'e', ENCODED => e99798d2c4d566ddb063efe03842457f,} transitioned from {testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,d,1365336962398.e99798d2c4d566ddb063efe03842457f. state=OPENING, ts=1365336963097, server=asf001.sp2.ygridcore.net,49518,1365336962323} to {testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,d,1365336962398.e99798d2c4d566ddb063efe03842457f. state=OPENING, ts=1365336963184, server=asf001.sp2.ygridcore.net,49518,1365336962323} 2013-04-07 12:16:03,185 INFO [hbase-am-zkevent-worker-pool-2-thread-35] master.RegionStates(264): Region {NAME => 'testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,,1365336962397.d4e129e51e67d798e17083623fdde133.', STARTKEY => '', ENDKEY => 'a', ENCODED => d4e129e51e67d798e17083623fdde133,} transitioned from {testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,,1365336962397.d4e129e51e67d798e17083623fdde133. state=OPENING, ts=1365336963112, server=asf001.sp2.ygridcore.net,43802,1365336889418} to {testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,,1365336962397.d4e129e51e67d798e17083623fdde133. state=OPEN, ts=1365336963185, server=asf001.sp2.ygridcore.net,43802,1365336889418} 2013-04-07 12:16:03,185 INFO [hbase-am-zkevent-worker-pool-2-thread-34] master.RegionStates(264): Region {NAME => 'testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,h,1365336962398.1ce6d62427641e2a0e135d3d27b54b5d.', STARTKEY => 'h', ENDKEY => 'i', ENCODED => 1ce6d62427641e2a0e135d3d27b54b5d,} transitioned from {testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,h,1365336962398.1ce6d62427641e2a0e135d3d27b54b5d. state=OPENING, ts=1365336963089, server=asf001.sp2.ygridcore.net,43802,1365336889418} to {testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,h,1365336962398.1ce6d62427641e2a0e135d3d27b54b5d. state=OPENING, ts=1365336963185, server=asf001.sp2.ygridcore.net,43802,1365336889418} 2013-04-07 12:16:03,185 DEBUG [Thread-200-EventThread] zookeeper.ZooKeeperWatcher(274): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/region-in-transition/1ce6d62427641e2a0e135d3d27b54b5d 2013-04-07 12:16:03,186 INFO [hbase-am-zkevent-worker-pool-2-thread-37] master.RegionStates(264): Region {NAME => 'testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,g,1365336962398.682475e1358b7306bd3406e9330263a4.', STARTKEY => 'g', ENDKEY => 'h', ENCODED => 682475e1358b7306bd3406e9330263a4,} transitioned from {testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,g,1365336962398.682475e1358b7306bd3406e9330263a4. state=OPENING, ts=1365336963119, server=asf001.sp2.ygridcore.net,49518,1365336962323} to {testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,g,1365336962398.682475e1358b7306bd3406e9330263a4. state=OPENING, ts=1365336963186, server=asf001.sp2.ygridcore.net,49518,1365336962323} 2013-04-07 12:16:03,186 DEBUG [RS_OPEN_REGION-asf001.sp2.ygridcore.net,60884,1365336878389-1] zookeeper.ZKUtil(1680): regionserver:60884-0x13de46b5f8b0001 Retrieved 156 byte(s) of data from znode /hbase/region-in-transition/11c8123e8d563f264d947ffa106e4c7c; data=region=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,b,1365336962398.11c8123e8d563f264d947ffa106e4c7c., state=RS_ZK_REGION_OPENING, servername=asf001.sp2.ygridcore.net,60884,1365336878389, createTime=1365336963170, payload.length=0 2013-04-07 12:16:03,187 INFO [hbase-am-zkevent-worker-pool-2-thread-39] master.RegionStates(264): Region {NAME => 'testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,f,1365336962398.eff8376859f5e6ba3201ec0ec9536d0f.', STARTKEY => 'f', ENDKEY => 'g', ENCODED => eff8376859f5e6ba3201ec0ec9536d0f,} transitioned from {testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,f,1365336962398.eff8376859f5e6ba3201ec0ec9536d0f. state=OPENING, ts=1365336963149, server=asf001.sp2.ygridcore.net,60884,1365336878389} to {testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,f,1365336962398.eff8376859f5e6ba3201ec0ec9536d0f. state=OPEN, ts=1365336963187, server=asf001.sp2.ygridcore.net,60884,1365336878389} 2013-04-07 12:16:03,189 DEBUG [Thread-200-EventThread] zookeeper.ZooKeeperWatcher(274): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/region-in-transition/11c8123e8d563f264d947ffa106e4c7c 2013-04-07 12:16:03,190 DEBUG [Thread-200-EventThread] zookeeper.ZooKeeperWatcher(274): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/region-in-transition/b1b340dff1e06fd6cb0b0a9c64964551 2013-04-07 12:16:03,190 DEBUG [Thread-200-EventThread] zookeeper.ZooKeeperWatcher(274): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/region-in-transition/385e5846630b0cb4493758961a2c2457 2013-04-07 12:16:03,191 DEBUG [Thread-200-EventThread] zookeeper.ZooKeeperWatcher(274): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/region-in-transition/d4e129e51e67d798e17083623fdde133 2013-04-07 12:16:03,191 DEBUG [hbase-am-zkevent-worker-pool-20-thread-31] master.AssignmentManager$4(1159): The znode of region testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,,1365336962397.d4e129e51e67d798e17083623fdde133. has been deleted. 2013-04-07 12:16:03,191 DEBUG [hbase-am-zkevent-worker-pool-2-thread-31] master.AssignmentManager$4(1159): The znode of region testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,,1365336962397.d4e129e51e67d798e17083623fdde133. has been deleted. 2013-04-07 12:16:03,191 INFO [hbase-am-zkevent-worker-pool-20-thread-31] master.RegionStates(264): Region {NAME => 'testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,,1365336962397.d4e129e51e67d798e17083623fdde133.', STARTKEY => '', ENDKEY => 'a', ENCODED => d4e129e51e67d798e17083623fdde133,} transitioned from {testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,,1365336962397.d4e129e51e67d798e17083623fdde133. state=OPEN, ts=1365336963151, server=asf001.sp2.ygridcore.net,43802,1365336889418} to {testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,,1365336962397.d4e129e51e67d798e17083623fdde133. state=OPEN, ts=1365336963191, server=asf001.sp2.ygridcore.net,43802,1365336889418} 2013-04-07 12:16:03,191 INFO [hbase-am-zkevent-worker-pool-2-thread-31] master.RegionStates(264): Region {NAME => 'testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,,1365336962397.d4e129e51e67d798e17083623fdde133.', STARTKEY => '', ENDKEY => 'a', ENCODED => d4e129e51e67d798e17083623fdde133,} transitioned from {testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,,1365336962397.d4e129e51e67d798e17083623fdde133. state=OPEN, ts=1365336963185, server=asf001.sp2.ygridcore.net,43802,1365336889418} to {testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,,1365336962397.d4e129e51e67d798e17083623fdde133. state=OPEN, ts=1365336963191, server=asf001.sp2.ygridcore.net,43802,1365336889418} 2013-04-07 12:16:03,191 INFO [hbase-am-zkevent-worker-pool-20-thread-31] master.RegionStates(299): Onlined region {NAME => 'testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,,1365336962397.d4e129e51e67d798e17083623fdde133.', STARTKEY => '', ENDKEY => 'a', ENCODED => d4e129e51e67d798e17083623fdde133,} on asf001.sp2.ygridcore.net,43802,1365336889418 2013-04-07 12:16:03,192 INFO [hbase-am-zkevent-worker-pool-2-thread-31] master.RegionStates(299): Onlined region {NAME => 'testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,,1365336962397.d4e129e51e67d798e17083623fdde133.', STARTKEY => '', ENDKEY => 'a', ENCODED => d4e129e51e67d798e17083623fdde133,} on asf001.sp2.ygridcore.net,43802,1365336889418 2013-04-07 12:16:03,192 INFO [hbase-am-zkevent-worker-pool-20-thread-31] master.AssignmentManager$4(1164): The master has opened the region testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,,1365336962397.d4e129e51e67d798e17083623fdde133. that was online on asf001.sp2.ygridcore.net,43802,1365336889418 2013-04-07 12:16:03,192 INFO [hbase-am-zkevent-worker-pool-2-thread-31] master.AssignmentManager$4(1164): The master has opened the region testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,,1365336962397.d4e129e51e67d798e17083623fdde133. that was online on asf001.sp2.ygridcore.net,43802,1365336889418 2013-04-07 12:16:03,191 DEBUG [Thread-200-EventThread] zookeeper.ZooKeeperWatcher(274): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/region-in-transition 2013-04-07 12:16:03,193 DEBUG [MASTER_OPEN_REGION-asf001.sp2.ygridcore.net,33796,1365336878036-2] zookeeper.ZKAssign(434): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Successfully deleted unassigned node for region d4e129e51e67d798e17083623fdde133 in expected state RS_ZK_REGION_OPENED 2013-04-07 12:16:03,193 DEBUG [MASTER_OPEN_REGION-asf001.sp2.ygridcore.net,33796,1365336878036-2] handler.OpenedRegionHandler(146): Handling OPENED event for testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,f,1365336962398.eff8376859f5e6ba3201ec0ec9536d0f. from asf001.sp2.ygridcore.net,60884,1365336878389; deleting unassigned node 2013-04-07 12:16:03,193 DEBUG [MASTER_OPEN_REGION-asf001.sp2.ygridcore.net,33796,1365336878036-2] zookeeper.ZKAssign(405): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Deleting existing unassigned node for eff8376859f5e6ba3201ec0ec9536d0f that is in expected state RS_ZK_REGION_OPENED 2013-04-07 12:16:03,193 DEBUG [Thread-200-EventThread] zookeeper.ZooKeeperWatcher(274): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/region-in-transition/38f0eb146fb473b19541deaada3419b9 2013-04-07 12:16:03,194 DEBUG [hbase-am-zkevent-worker-pool-2-thread-40] zookeeper.ZKUtil(1680): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Retrieved 156 byte(s) of data from znode /hbase/region-in-transition/dbb82f7f432d6664327da0b8f15b7d76 and set watcher; region=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,e,1365336962398.dbb82f7f432d6664327da0b8f15b7d76., state=RS_ZK_REGION_OPENED, servername=asf001.sp2.ygridcore.net,60884,1365336878389, createTime=1365336963155, payload.length=0 2013-04-07 12:16:03,194 DEBUG [hbase-am-zkevent-worker-pool-2-thread-40] master.AssignmentManager(740): Handling transition=RS_ZK_REGION_OPENED, server=asf001.sp2.ygridcore.net,60884,1365336878389, region=dbb82f7f432d6664327da0b8f15b7d76, current state from region state map ={testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,e,1365336962398.dbb82f7f432d6664327da0b8f15b7d76. state=OPENING, ts=1365336963171, server=asf001.sp2.ygridcore.net,60884,1365336878389} 2013-04-07 12:16:03,196 INFO [hbase-am-zkevent-worker-pool-2-thread-40] master.RegionStates(264): Region {NAME => 'testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,e,1365336962398.dbb82f7f432d6664327da0b8f15b7d76.', STARTKEY => 'e', ENDKEY => 'f', ENCODED => dbb82f7f432d6664327da0b8f15b7d76,} transitioned from {testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,e,1365336962398.dbb82f7f432d6664327da0b8f15b7d76. state=OPENING, ts=1365336963171, server=asf001.sp2.ygridcore.net,60884,1365336878389} to {testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,e,1365336962398.dbb82f7f432d6664327da0b8f15b7d76. state=OPEN, ts=1365336963196, server=asf001.sp2.ygridcore.net,60884,1365336878389} 2013-04-07 12:16:03,197 DEBUG [Thread-200-EventThread] zookeeper.ZooKeeperWatcher(274): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/region-in-transition/4432e800611c00a4ae168862a273fc70 2013-04-07 12:16:03,197 DEBUG [hbase-am-zkevent-worker-pool-20-thread-34] master.AssignmentManager$4(1159): The znode of region testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,c,1365336962398.4432e800611c00a4ae168862a273fc70. has been deleted. 2013-04-07 12:16:03,197 INFO [hbase-am-zkevent-worker-pool-20-thread-34] master.RegionStates(264): Region {NAME => 'testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,c,1365336962398.4432e800611c00a4ae168862a273fc70.', STARTKEY => 'c', ENDKEY => 'd', ENCODED => 4432e800611c00a4ae168862a273fc70,} transitioned from {testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,c,1365336962398.4432e800611c00a4ae168862a273fc70. state=OPEN, ts=1365336963156, server=asf001.sp2.ygridcore.net,43802,1365336889418} to {testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,c,1365336962398.4432e800611c00a4ae168862a273fc70. state=OPEN, ts=1365336963197, server=asf001.sp2.ygridcore.net,43802,1365336889418} 2013-04-07 12:16:03,197 INFO [hbase-am-zkevent-worker-pool-20-thread-34] master.RegionStates(299): Onlined region {NAME => 'testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,c,1365336962398.4432e800611c00a4ae168862a273fc70.', STARTKEY => 'c', ENDKEY => 'd', ENCODED => 4432e800611c00a4ae168862a273fc70,} on asf001.sp2.ygridcore.net,43802,1365336889418 2013-04-07 12:16:03,198 INFO [hbase-am-zkevent-worker-pool-20-thread-34] master.AssignmentManager$4(1164): The master has opened the region testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,c,1365336962398.4432e800611c00a4ae168862a273fc70. that was online on asf001.sp2.ygridcore.net,43802,1365336889418 2013-04-07 12:16:03,198 INFO [PostOpenDeployTasks:11c8123e8d563f264d947ffa106e4c7c] regionserver.HRegionServer(1563): Post open deploy tasks for region=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,b,1365336962398.11c8123e8d563f264d947ffa106e4c7c. 2013-04-07 12:16:03,198 DEBUG [MASTER_OPEN_REGION-asf001.sp2.ygridcore.net,33796,1365336878036-3] zookeeper.ZKAssign(434): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Successfully deleted unassigned node for region 4432e800611c00a4ae168862a273fc70 in expected state RS_ZK_REGION_OPENED 2013-04-07 12:16:03,198 DEBUG [MASTER_OPEN_REGION-asf001.sp2.ygridcore.net,33796,1365336878036-3] handler.OpenedRegionHandler(146): Handling OPENED event for testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,e,1365336962398.dbb82f7f432d6664327da0b8f15b7d76. from asf001.sp2.ygridcore.net,60884,1365336878389; deleting unassigned node 2013-04-07 12:16:03,198 DEBUG [MASTER_OPEN_REGION-asf001.sp2.ygridcore.net,33796,1365336878036-3] zookeeper.ZKAssign(405): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Deleting existing unassigned node for dbb82f7f432d6664327da0b8f15b7d76 that is in expected state RS_ZK_REGION_OPENED 2013-04-07 12:16:03,199 DEBUG [MASTER_OPEN_REGION-asf001.sp2.ygridcore.net,33796,1365336878036-1] zookeeper.ZKUtil(1680): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Retrieved 156 byte(s) of data from znode /hbase/region-in-transition/eff8376859f5e6ba3201ec0ec9536d0f; data=region=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,f,1365336962398.eff8376859f5e6ba3201ec0ec9536d0f., state=RS_ZK_REGION_OPENED, servername=asf001.sp2.ygridcore.net,60884,1365336878389, createTime=1365336963154, payload.length=0 2013-04-07 12:16:03,199 DEBUG [PostOpenDeployTasks:11c8123e8d563f264d947ffa106e4c7c] zookeeper.ZKUtil(1680): hconnection-0x43ad4a-0x13de46b5f8b0005 Retrieved 43 byte(s) of data from znode /hbase/meta-region-server; data=asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:16:03,200 DEBUG [hbase-am-zkevent-worker-pool-2-thread-32] zookeeper.ZKUtil(1680): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Retrieved 156 byte(s) of data from znode /hbase/region-in-transition/b1b340dff1e06fd6cb0b0a9c64964551 and set watcher; region=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,a,1365336962398.b1b340dff1e06fd6cb0b0a9c64964551., state=RS_ZK_REGION_OPENED, servername=asf001.sp2.ygridcore.net,49518,1365336962323, createTime=1365336963169, payload.length=0 2013-04-07 12:16:03,200 DEBUG [hbase-am-zkevent-worker-pool-20-thread-25] zookeeper.ZKUtil(1680): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Retrieved 156 byte(s) of data from znode /hbase/region-in-transition/682475e1358b7306bd3406e9330263a4 and set watcher; region=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,g,1365336962398.682475e1358b7306bd3406e9330263a4., state=RS_ZK_REGION_OPENED, servername=asf001.sp2.ygridcore.net,49518,1365336962323, createTime=1365336963154, payload.length=0 2013-04-07 12:16:03,201 DEBUG [hbase-am-zkevent-worker-pool-2-thread-32] master.AssignmentManager(740): Handling transition=RS_ZK_REGION_OPENED, server=asf001.sp2.ygridcore.net,49518,1365336962323, region=b1b340dff1e06fd6cb0b0a9c64964551, current state from region state map ={testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,a,1365336962398.b1b340dff1e06fd6cb0b0a9c64964551. state=OPENING, ts=1365336963180, server=asf001.sp2.ygridcore.net,49518,1365336962323} 2013-04-07 12:16:03,201 DEBUG [hbase-am-zkevent-worker-pool-20-thread-25] master.AssignmentManager(740): Handling transition=RS_ZK_REGION_OPENED, server=asf001.sp2.ygridcore.net,49518,1365336962323, region=682475e1358b7306bd3406e9330263a4, current state from region state map ={testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,g,1365336962398.682475e1358b7306bd3406e9330263a4. state=OPENING, ts=1365336963153, server=asf001.sp2.ygridcore.net,49518,1365336962323} 2013-04-07 12:16:03,201 INFO [hbase-am-zkevent-worker-pool-2-thread-32] master.RegionStates(264): Region {NAME => 'testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,a,1365336962398.b1b340dff1e06fd6cb0b0a9c64964551.', STARTKEY => 'a', ENDKEY => 'b', ENCODED => b1b340dff1e06fd6cb0b0a9c64964551,} transitioned from {testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,a,1365336962398.b1b340dff1e06fd6cb0b0a9c64964551. state=OPENING, ts=1365336963180, server=asf001.sp2.ygridcore.net,49518,1365336962323} to {testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,a,1365336962398.b1b340dff1e06fd6cb0b0a9c64964551. state=OPEN, ts=1365336963201, server=asf001.sp2.ygridcore.net,49518,1365336962323} 2013-04-07 12:16:03,201 INFO [hbase-am-zkevent-worker-pool-20-thread-25] master.RegionStates(264): Region {NAME => 'testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,g,1365336962398.682475e1358b7306bd3406e9330263a4.', STARTKEY => 'g', ENDKEY => 'h', ENCODED => 682475e1358b7306bd3406e9330263a4,} transitioned from {testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,g,1365336962398.682475e1358b7306bd3406e9330263a4. state=OPENING, ts=1365336963153, server=asf001.sp2.ygridcore.net,49518,1365336962323} to {testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,g,1365336962398.682475e1358b7306bd3406e9330263a4. state=OPEN, ts=1365336963201, server=asf001.sp2.ygridcore.net,49518,1365336962323} 2013-04-07 12:16:03,201 DEBUG [hbase-am-zkevent-worker-pool-20-thread-27] zookeeper.ZKUtil(1680): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Retrieved 156 byte(s) of data from znode /hbase/region-in-transition/e99798d2c4d566ddb063efe03842457f and set watcher; region=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,d,1365336962398.e99798d2c4d566ddb063efe03842457f., state=RS_ZK_REGION_OPENED, servername=asf001.sp2.ygridcore.net,49518,1365336962323, createTime=1365336963163, payload.length=0 2013-04-07 12:16:03,202 DEBUG [hbase-am-zkevent-worker-pool-20-thread-27] master.AssignmentManager(740): Handling transition=RS_ZK_REGION_OPENED, server=asf001.sp2.ygridcore.net,49518,1365336962323, region=e99798d2c4d566ddb063efe03842457f, current state from region state map ={testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,d,1365336962398.e99798d2c4d566ddb063efe03842457f. state=OPENING, ts=1365336963151, server=asf001.sp2.ygridcore.net,49518,1365336962323} 2013-04-07 12:16:03,201 DEBUG [MASTER_OPEN_REGION-asf001.sp2.ygridcore.net,33796,1365336878036-4] zookeeper.ZKUtil(1680): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Retrieved 156 byte(s) of data from znode /hbase/region-in-transition/dbb82f7f432d6664327da0b8f15b7d76; data=region=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,e,1365336962398.dbb82f7f432d6664327da0b8f15b7d76., state=RS_ZK_REGION_OPENED, servername=asf001.sp2.ygridcore.net,60884,1365336878389, createTime=1365336963155, payload.length=0 2013-04-07 12:16:03,202 INFO [hbase-am-zkevent-worker-pool-20-thread-27] master.RegionStates(264): Region {NAME => 'testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,d,1365336962398.e99798d2c4d566ddb063efe03842457f.', STARTKEY => 'd', ENDKEY => 'e', ENCODED => e99798d2c4d566ddb063efe03842457f,} transitioned from {testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,d,1365336962398.e99798d2c4d566ddb063efe03842457f. state=OPENING, ts=1365336963151, server=asf001.sp2.ygridcore.net,49518,1365336962323} to {testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,d,1365336962398.e99798d2c4d566ddb063efe03842457f. state=OPEN, ts=1365336963202, server=asf001.sp2.ygridcore.net,49518,1365336962323} 2013-04-07 12:16:03,203 DEBUG [hbase-am-zkevent-worker-pool-2-thread-33] master.AssignmentManager$4(1159): The znode of region testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,c,1365336962398.4432e800611c00a4ae168862a273fc70. has been deleted. 2013-04-07 12:16:03,204 DEBUG [hbase-am-zkevent-worker-pool-2-thread-36] zookeeper.ZKUtil(1680): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Retrieved 156 byte(s) of data from znode /hbase/region-in-transition/e99798d2c4d566ddb063efe03842457f and set watcher; region=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,d,1365336962398.e99798d2c4d566ddb063efe03842457f., state=RS_ZK_REGION_OPENED, servername=asf001.sp2.ygridcore.net,49518,1365336962323, createTime=1365336963163, payload.length=0 2013-04-07 12:16:03,204 DEBUG [hbase-am-zkevent-worker-pool-2-thread-36] master.AssignmentManager(740): Handling transition=RS_ZK_REGION_OPENED, server=asf001.sp2.ygridcore.net,49518,1365336962323, region=e99798d2c4d566ddb063efe03842457f, current state from region state map ={testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,d,1365336962398.e99798d2c4d566ddb063efe03842457f. state=OPENING, ts=1365336963184, server=asf001.sp2.ygridcore.net,49518,1365336962323} 2013-04-07 12:16:03,204 INFO [hbase-am-zkevent-worker-pool-2-thread-36] master.RegionStates(264): Region {NAME => 'testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,d,1365336962398.e99798d2c4d566ddb063efe03842457f.', STARTKEY => 'd', ENDKEY => 'e', ENCODED => e99798d2c4d566ddb063efe03842457f,} transitioned from {testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,d,1365336962398.e99798d2c4d566ddb063efe03842457f. state=OPENING, ts=1365336963184, server=asf001.sp2.ygridcore.net,49518,1365336962323} to {testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,d,1365336962398.e99798d2c4d566ddb063efe03842457f. state=OPEN, ts=1365336963204, server=asf001.sp2.ygridcore.net,49518,1365336962323} 2013-04-07 12:16:03,204 DEBUG [MASTER_OPEN_REGION-asf001.sp2.ygridcore.net,33796,1365336878036-0] zookeeper.ZKUtil(775): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Unable to get data of znode /hbase/region-in-transition/d4e129e51e67d798e17083623fdde133 because node does not exist (not necessarily an error) 2013-04-07 12:16:03,205 WARN [MASTER_OPEN_REGION-asf001.sp2.ygridcore.net,33796,1365336878036-0] handler.OpenedRegionHandler(132): The znode of the region testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,,1365336962397.d4e129e51e67d798e17083623fdde133. would have already been deleted 2013-04-07 12:16:03,205 ERROR [MASTER_OPEN_REGION-asf001.sp2.ygridcore.net,33796,1365336878036-0] handler.OpenedRegionHandler(104): The znode of region testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,,1365336962397.d4e129e51e67d798e17083623fdde133. could not be deleted. 2013-04-07 12:16:03,206 DEBUG [MASTER_OPEN_REGION-asf001.sp2.ygridcore.net,33796,1365336878036-0] handler.OpenedRegionHandler(146): Handling OPENED event for testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,a,1365336962398.b1b340dff1e06fd6cb0b0a9c64964551. from asf001.sp2.ygridcore.net,49518,1365336962323; deleting unassigned node 2013-04-07 12:16:03,206 DEBUG [MASTER_OPEN_REGION-asf001.sp2.ygridcore.net,33796,1365336878036-0] zookeeper.ZKAssign(405): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Deleting existing unassigned node for b1b340dff1e06fd6cb0b0a9c64964551 that is in expected state RS_ZK_REGION_OPENED 2013-04-07 12:16:03,207 DEBUG [hbase-am-zkevent-worker-pool-2-thread-39] zookeeper.ZKUtil(1680): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Retrieved 156 byte(s) of data from znode /hbase/region-in-transition/eff8376859f5e6ba3201ec0ec9536d0f and set watcher; region=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,f,1365336962398.eff8376859f5e6ba3201ec0ec9536d0f., state=RS_ZK_REGION_OPENED, servername=asf001.sp2.ygridcore.net,60884,1365336878389, createTime=1365336963154, payload.length=0 2013-04-07 12:16:03,207 DEBUG [hbase-am-zkevent-worker-pool-2-thread-39] master.AssignmentManager(740): Handling transition=RS_ZK_REGION_OPENED, server=asf001.sp2.ygridcore.net,60884,1365336878389, region=eff8376859f5e6ba3201ec0ec9536d0f, current state from region state map ={testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,f,1365336962398.eff8376859f5e6ba3201ec0ec9536d0f. state=OPEN, ts=1365336963187, server=asf001.sp2.ygridcore.net,60884,1365336878389} 2013-04-07 12:16:03,208 WARN [hbase-am-zkevent-worker-pool-2-thread-39] master.AssignmentManager(934): Received OPENED for region eff8376859f5e6ba3201ec0ec9536d0f from server asf001.sp2.ygridcore.net,60884,1365336878389 but region was in the state {testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,f,1365336962398.eff8376859f5e6ba3201ec0ec9536d0f. state=OPEN, ts=1365336963187, server=asf001.sp2.ygridcore.net,60884,1365336878389} and not in expected PENDING_OPEN or OPENING states, or not on the expected server 2013-04-07 12:16:03,208 DEBUG [hbase-am-zkevent-worker-pool-2-thread-27] zookeeper.ZKUtil(1680): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Retrieved 156 byte(s) of data from znode /hbase/region-in-transition/682475e1358b7306bd3406e9330263a4 and set watcher; region=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,g,1365336962398.682475e1358b7306bd3406e9330263a4., state=RS_ZK_REGION_OPENED, servername=asf001.sp2.ygridcore.net,49518,1365336962323, createTime=1365336963154, payload.length=0 2013-04-07 12:16:03,208 INFO [PostOpenDeployTasks:11c8123e8d563f264d947ffa106e4c7c] catalog.MetaEditor(432): Updated row testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,b,1365336962398.11c8123e8d563f264d947ffa106e4c7c. with server=asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:16:03,208 DEBUG [hbase-am-zkevent-worker-pool-2-thread-27] master.AssignmentManager(740): Handling transition=RS_ZK_REGION_OPENED, server=asf001.sp2.ygridcore.net,49518,1365336962323, region=682475e1358b7306bd3406e9330263a4, current state from region state map ={testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,g,1365336962398.682475e1358b7306bd3406e9330263a4. state=OPENING, ts=1365336963186, server=asf001.sp2.ygridcore.net,49518,1365336962323} 2013-04-07 12:16:03,208 DEBUG [hbase-am-zkevent-worker-pool-2-thread-37] zookeeper.ZKUtil(1680): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Retrieved 156 byte(s) of data from znode /hbase/region-in-transition/1ce6d62427641e2a0e135d3d27b54b5d and set watcher; region=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,h,1365336962398.1ce6d62427641e2a0e135d3d27b54b5d., state=RS_ZK_REGION_OPENED, servername=asf001.sp2.ygridcore.net,43802,1365336889418, createTime=1365336963163, payload.length=0 2013-04-07 12:16:03,208 INFO [hbase-am-zkevent-worker-pool-2-thread-27] master.RegionStates(264): Region {NAME => 'testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,g,1365336962398.682475e1358b7306bd3406e9330263a4.', STARTKEY => 'g', ENDKEY => 'h', ENCODED => 682475e1358b7306bd3406e9330263a4,} transitioned from {testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,g,1365336962398.682475e1358b7306bd3406e9330263a4. state=OPENING, ts=1365336963186, server=asf001.sp2.ygridcore.net,49518,1365336962323} to {testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,g,1365336962398.682475e1358b7306bd3406e9330263a4. state=OPEN, ts=1365336963208, server=asf001.sp2.ygridcore.net,49518,1365336962323} 2013-04-07 12:16:03,208 INFO [PRI IPC Server handler 6 on 60884] regionserver.HRegionServer(3434): Received close region: eff8376859f5e6ba3201ec0ec9536d0fTransitioning in ZK: no. Version of ZK closing node:-1. Destination server:null 2013-04-07 12:16:03,208 INFO [PostOpenDeployTasks:11c8123e8d563f264d947ffa106e4c7c] regionserver.HRegionServer(1584): Done with post open deploy task for region=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,b,1365336962398.11c8123e8d563f264d947ffa106e4c7c. 2013-04-07 12:16:03,209 DEBUG [hbase-am-zkevent-worker-pool-2-thread-37] master.AssignmentManager(740): Handling transition=RS_ZK_REGION_OPENED, server=asf001.sp2.ygridcore.net,43802,1365336889418, region=1ce6d62427641e2a0e135d3d27b54b5d, current state from region state map ={testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,h,1365336962398.1ce6d62427641e2a0e135d3d27b54b5d. state=OPENING, ts=1365336963185, server=asf001.sp2.ygridcore.net,43802,1365336889418} 2013-04-07 12:16:03,209 INFO [hbase-am-zkevent-worker-pool-2-thread-37] master.RegionStates(264): Region {NAME => 'testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,h,1365336962398.1ce6d62427641e2a0e135d3d27b54b5d.', STARTKEY => 'h', ENDKEY => 'i', ENCODED => 1ce6d62427641e2a0e135d3d27b54b5d,} transitioned from {testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,h,1365336962398.1ce6d62427641e2a0e135d3d27b54b5d. state=OPENING, ts=1365336963185, server=asf001.sp2.ygridcore.net,43802,1365336889418} to {testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,h,1365336962398.1ce6d62427641e2a0e135d3d27b54b5d. state=OPEN, ts=1365336963209, server=asf001.sp2.ygridcore.net,43802,1365336889418} 2013-04-07 12:16:03,210 DEBUG [hbase-am-zkevent-worker-pool-20-thread-28] zookeeper.ZKUtil(1680): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Retrieved 156 byte(s) of data from znode /hbase/region-in-transition/1ce6d62427641e2a0e135d3d27b54b5d and set watcher; region=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,h,1365336962398.1ce6d62427641e2a0e135d3d27b54b5d., state=RS_ZK_REGION_OPENED, servername=asf001.sp2.ygridcore.net,43802,1365336889418, createTime=1365336963163, payload.length=0 2013-04-07 12:16:03,210 DEBUG [hbase-am-zkevent-worker-pool-20-thread-28] master.AssignmentManager(740): Handling transition=RS_ZK_REGION_OPENED, server=asf001.sp2.ygridcore.net,43802,1365336889418, region=1ce6d62427641e2a0e135d3d27b54b5d, current state from region state map ={testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,h,1365336962398.1ce6d62427641e2a0e135d3d27b54b5d. state=OPENING, ts=1365336963145, server=asf001.sp2.ygridcore.net,43802,1365336889418} 2013-04-07 12:16:03,210 INFO [hbase-am-zkevent-worker-pool-20-thread-28] master.RegionStates(264): Region {NAME => 'testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,h,1365336962398.1ce6d62427641e2a0e135d3d27b54b5d.', STARTKEY => 'h', ENDKEY => 'i', ENCODED => 1ce6d62427641e2a0e135d3d27b54b5d,} transitioned from {testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,h,1365336962398.1ce6d62427641e2a0e135d3d27b54b5d. state=OPENING, ts=1365336963145, server=asf001.sp2.ygridcore.net,43802,1365336889418} to {testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,h,1365336962398.1ce6d62427641e2a0e135d3d27b54b5d. state=OPEN, ts=1365336963210, server=asf001.sp2.ygridcore.net,43802,1365336889418} 2013-04-07 12:16:03,210 DEBUG [RS_CLOSE_REGION-asf001.sp2.ygridcore.net,60884,1365336878389-1] handler.CloseRegionHandler(125): Processing close of testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,f,1365336962398.eff8376859f5e6ba3201ec0ec9536d0f. 2013-04-07 12:16:03,210 DEBUG [hbase-am-zkevent-worker-pool-2-thread-39] master.AssignmentManager(1638): Sent CLOSE to asf001.sp2.ygridcore.net,60884,1365336878389 for region testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,f,1365336962398.eff8376859f5e6ba3201ec0ec9536d0f. 2013-04-07 12:16:03,211 DEBUG [RS_OPEN_REGION-asf001.sp2.ygridcore.net,60884,1365336878389-1] zookeeper.ZKAssign(721): regionserver:60884-0x13de46b5f8b0001 Attempting to transition node 11c8123e8d563f264d947ffa106e4c7c from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENED 2013-04-07 12:16:03,212 DEBUG [hbase-am-zkevent-worker-pool-20-thread-26] zookeeper.ZKUtil(1680): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Retrieved 156 byte(s) of data from znode /hbase/region-in-transition/11c8123e8d563f264d947ffa106e4c7c and set watcher; region=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,b,1365336962398.11c8123e8d563f264d947ffa106e4c7c., state=RS_ZK_REGION_OPENING, servername=asf001.sp2.ygridcore.net,60884,1365336878389, createTime=1365336963187, payload.length=0 2013-04-07 12:16:03,213 DEBUG [hbase-am-zkevent-worker-pool-20-thread-26] master.AssignmentManager(740): Handling transition=RS_ZK_REGION_OPENING, server=asf001.sp2.ygridcore.net,60884,1365336878389, region=11c8123e8d563f264d947ffa106e4c7c, current state from region state map ={testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,b,1365336962398.11c8123e8d563f264d947ffa106e4c7c. state=PENDING_OPEN, ts=1365336963018, server=asf001.sp2.ygridcore.net,60884,1365336878389} 2013-04-07 12:16:03,213 INFO [hbase-am-zkevent-worker-pool-20-thread-26] master.RegionStates(264): Region {NAME => 'testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,b,1365336962398.11c8123e8d563f264d947ffa106e4c7c.', STARTKEY => 'b', ENDKEY => 'c', ENCODED => 11c8123e8d563f264d947ffa106e4c7c,} transitioned from {testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,b,1365336962398.11c8123e8d563f264d947ffa106e4c7c. state=PENDING_OPEN, ts=1365336963018, server=asf001.sp2.ygridcore.net,60884,1365336878389} to {testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,b,1365336962398.11c8123e8d563f264d947ffa106e4c7c. state=OPENING, ts=1365336963213, server=asf001.sp2.ygridcore.net,60884,1365336878389} 2013-04-07 12:16:03,213 DEBUG [RS_CLOSE_REGION-asf001.sp2.ygridcore.net,60884,1365336878389-1] regionserver.HRegion(889): Closing testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,f,1365336962398.eff8376859f5e6ba3201ec0ec9536d0f.: disabling compactions & flushes 2013-04-07 12:16:03,214 DEBUG [RS_CLOSE_REGION-asf001.sp2.ygridcore.net,60884,1365336878389-1] regionserver.HRegion(911): Updates disabled for region testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,f,1365336962398.eff8376859f5e6ba3201ec0ec9536d0f. 2013-04-07 12:16:03,214 INFO [RS_CLOSE_REGION-asf001.sp2.ygridcore.net,60884,1365336878389-1] regionserver.HRegion(968): Closed testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,f,1365336962398.eff8376859f5e6ba3201ec0ec9536d0f. 2013-04-07 12:16:03,214 DEBUG [RS_CLOSE_REGION-asf001.sp2.ygridcore.net,60884,1365336878389-1] handler.CloseRegionHandler(177): Closed region testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,f,1365336962398.eff8376859f5e6ba3201ec0ec9536d0f. 2013-04-07 12:16:03,214 DEBUG [RS_OPEN_REGION-asf001.sp2.ygridcore.net,60884,1365336878389-1] zookeeper.ZKUtil(1680): regionserver:60884-0x13de46b5f8b0001 Retrieved 156 byte(s) of data from znode /hbase/region-in-transition/11c8123e8d563f264d947ffa106e4c7c; data=region=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,b,1365336962398.11c8123e8d563f264d947ffa106e4c7c., state=RS_ZK_REGION_OPENING, servername=asf001.sp2.ygridcore.net,60884,1365336878389, createTime=1365336963187, payload.length=0 2013-04-07 12:16:03,215 DEBUG [hbase-am-zkevent-worker-pool-2-thread-22] zookeeper.ZKUtil(1680): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Retrieved 156 byte(s) of data from znode /hbase/region-in-transition/11c8123e8d563f264d947ffa106e4c7c and set watcher; region=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,b,1365336962398.11c8123e8d563f264d947ffa106e4c7c., state=RS_ZK_REGION_OPENING, servername=asf001.sp2.ygridcore.net,60884,1365336878389, createTime=1365336963187, payload.length=0 2013-04-07 12:16:03,215 DEBUG [hbase-am-zkevent-worker-pool-2-thread-22] master.AssignmentManager(740): Handling transition=RS_ZK_REGION_OPENING, server=asf001.sp2.ygridcore.net,60884,1365336878389, region=11c8123e8d563f264d947ffa106e4c7c, current state from region state map =null 2013-04-07 12:16:03,218 DEBUG [hbase-am-zkevent-worker-pool-20-thread-29] zookeeper.ZKUtil(1680): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Retrieved 156 byte(s) of data from znode /hbase/region-in-transition/b1b340dff1e06fd6cb0b0a9c64964551 and set watcher; region=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,a,1365336962398.b1b340dff1e06fd6cb0b0a9c64964551., state=RS_ZK_REGION_OPENED, servername=asf001.sp2.ygridcore.net,49518,1365336962323, createTime=1365336963169, payload.length=0 2013-04-07 12:16:03,218 DEBUG [hbase-am-zkevent-worker-pool-20-thread-29] master.AssignmentManager(740): Handling transition=RS_ZK_REGION_OPENED, server=asf001.sp2.ygridcore.net,49518,1365336962323, region=b1b340dff1e06fd6cb0b0a9c64964551, current state from region state map ={testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,a,1365336962398.b1b340dff1e06fd6cb0b0a9c64964551. state=OPENING, ts=1365336963153, server=asf001.sp2.ygridcore.net,49518,1365336962323} 2013-04-07 12:16:03,218 DEBUG [hbase-am-zkevent-worker-pool-2-thread-22] zookeeper.ZKUtil(1680): hconnection-0x1f327e-0x13de46b5f8b0003 Retrieved 43 byte(s) of data from znode /hbase/meta-region-server; data=asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:16:03,218 INFO [hbase-am-zkevent-worker-pool-20-thread-29] master.RegionStates(264): Region {NAME => 'testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,a,1365336962398.b1b340dff1e06fd6cb0b0a9c64964551.', STARTKEY => 'a', ENDKEY => 'b', ENCODED => b1b340dff1e06fd6cb0b0a9c64964551,} transitioned from {testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,a,1365336962398.b1b340dff1e06fd6cb0b0a9c64964551. state=OPENING, ts=1365336963153, server=asf001.sp2.ygridcore.net,49518,1365336962323} to {testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,a,1365336962398.b1b340dff1e06fd6cb0b0a9c64964551. state=OPEN, ts=1365336963218, server=asf001.sp2.ygridcore.net,49518,1365336962323} 2013-04-07 12:16:03,220 DEBUG [hbase-am-zkevent-worker-pool-20-thread-30] zookeeper.ZKUtil(1680): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Retrieved 156 byte(s) of data from znode /hbase/region-in-transition/385e5846630b0cb4493758961a2c2457 and set watcher; region=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,i,1365336962398.385e5846630b0cb4493758961a2c2457., state=RS_ZK_REGION_OPENED, servername=asf001.sp2.ygridcore.net,60884,1365336878389, createTime=1365336963171, payload.length=0 2013-04-07 12:16:03,220 DEBUG [hbase-am-zkevent-worker-pool-20-thread-30] master.AssignmentManager(740): Handling transition=RS_ZK_REGION_OPENED, server=asf001.sp2.ygridcore.net,60884,1365336878389, region=385e5846630b0cb4493758961a2c2457, current state from region state map ={testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,i,1365336962398.385e5846630b0cb4493758961a2c2457. state=OPENING, ts=1365336963154, server=asf001.sp2.ygridcore.net,60884,1365336878389} 2013-04-07 12:16:03,220 INFO [hbase-am-zkevent-worker-pool-20-thread-30] master.RegionStates(264): Region {NAME => 'testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,i,1365336962398.385e5846630b0cb4493758961a2c2457.', STARTKEY => 'i', ENDKEY => 'j', ENCODED => 385e5846630b0cb4493758961a2c2457,} transitioned from {testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,i,1365336962398.385e5846630b0cb4493758961a2c2457. state=OPENING, ts=1365336963154, server=asf001.sp2.ygridcore.net,60884,1365336878389} to {testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,i,1365336962398.385e5846630b0cb4493758961a2c2457. state=OPEN, ts=1365336963220, server=asf001.sp2.ygridcore.net,60884,1365336878389} 2013-04-07 12:16:03,220 DEBUG [hbase-am-zkevent-worker-pool-2-thread-22] zookeeper.ZKUtil(1680): hconnection-0x1f327e-0x13de46b5f8b0003 Retrieved 43 byte(s) of data from znode /hbase/meta-region-server; data=asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:16:03,220 DEBUG [hbase-am-zkevent-worker-pool-2-thread-28] zookeeper.ZKUtil(1680): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Retrieved 156 byte(s) of data from znode /hbase/region-in-transition/385e5846630b0cb4493758961a2c2457 and set watcher; region=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,i,1365336962398.385e5846630b0cb4493758961a2c2457., state=RS_ZK_REGION_OPENED, servername=asf001.sp2.ygridcore.net,60884,1365336878389, createTime=1365336963171, payload.length=0 2013-04-07 12:16:03,222 DEBUG [hbase-am-zkevent-worker-pool-20-thread-33] zookeeper.ZKUtil(1680): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Retrieved 156 byte(s) of data from znode /hbase/region-in-transition/38f0eb146fb473b19541deaada3419b9 and set watcher; region=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,j,1365336962398.38f0eb146fb473b19541deaada3419b9., state=RS_ZK_REGION_OPENING, servername=asf001.sp2.ygridcore.net,49518,1365336962323, createTime=1365336963173, payload.length=0 2013-04-07 12:16:03,222 DEBUG [hbase-am-zkevent-worker-pool-20-thread-33] master.AssignmentManager(740): Handling transition=RS_ZK_REGION_OPENING, server=asf001.sp2.ygridcore.net,49518,1365336962323, region=38f0eb146fb473b19541deaada3419b9, current state from region state map ={testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,j,1365336962398.38f0eb146fb473b19541deaada3419b9. state=PENDING_OPEN, ts=1365336963019, server=asf001.sp2.ygridcore.net,49518,1365336962323} 2013-04-07 12:16:03,222 INFO [hbase-am-zkevent-worker-pool-20-thread-33] master.RegionStates(264): Region {NAME => 'testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,j,1365336962398.38f0eb146fb473b19541deaada3419b9.', STARTKEY => 'j', ENDKEY => '', ENCODED => 38f0eb146fb473b19541deaada3419b9,} transitioned from {testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,j,1365336962398.38f0eb146fb473b19541deaada3419b9. state=PENDING_OPEN, ts=1365336963019, server=asf001.sp2.ygridcore.net,49518,1365336962323} to {testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,j,1365336962398.38f0eb146fb473b19541deaada3419b9. state=OPENING, ts=1365336963222, server=asf001.sp2.ygridcore.net,49518,1365336962323} 2013-04-07 12:16:03,222 INFO [hbase-am-zkevent-worker-pool-2-thread-22] master.RegionStates(264): Region {NAME => 'testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,b,1365336962398.11c8123e8d563f264d947ffa106e4c7c.', STARTKEY => 'b', ENDKEY => 'c', ENCODED => 11c8123e8d563f264d947ffa106e4c7c,} transitioned from {testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,b,1365336962398.11c8123e8d563f264d947ffa106e4c7c. state=OFFLINE, ts=1365336963222, server=null} to {testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,b,1365336962398.11c8123e8d563f264d947ffa106e4c7c. state=OPENING, ts=1365336963222, server=asf001.sp2.ygridcore.net,60884,1365336878389} 2013-04-07 12:16:03,223 DEBUG [hbase-am-zkevent-worker-pool-2-thread-28] master.AssignmentManager(740): Handling transition=RS_ZK_REGION_OPENED, server=asf001.sp2.ygridcore.net,60884,1365336878389, region=385e5846630b0cb4493758961a2c2457, current state from region state map ={testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,i,1365336962398.385e5846630b0cb4493758961a2c2457. state=OPENING, ts=1365336963181, server=asf001.sp2.ygridcore.net,60884,1365336878389} 2013-04-07 12:16:03,223 INFO [hbase-am-zkevent-worker-pool-2-thread-28] master.RegionStates(264): Region {NAME => 'testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,i,1365336962398.385e5846630b0cb4493758961a2c2457.', STARTKEY => 'i', ENDKEY => 'j', ENCODED => 385e5846630b0cb4493758961a2c2457,} transitioned from {testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,i,1365336962398.385e5846630b0cb4493758961a2c2457. state=OPENING, ts=1365336963181, server=asf001.sp2.ygridcore.net,60884,1365336878389} to {testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,i,1365336962398.385e5846630b0cb4493758961a2c2457. state=OPEN, ts=1365336963223, server=asf001.sp2.ygridcore.net,60884,1365336878389} 2013-04-07 12:16:03,223 DEBUG [Thread-200-EventThread] zookeeper.ZooKeeperWatcher(274): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/region-in-transition/38f0eb146fb473b19541deaada3419b9 2013-04-07 12:16:03,224 DEBUG [hbase-am-zkevent-worker-pool-2-thread-26] zookeeper.ZKUtil(1680): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Retrieved 156 byte(s) of data from znode /hbase/region-in-transition/38f0eb146fb473b19541deaada3419b9 and set watcher; region=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,j,1365336962398.38f0eb146fb473b19541deaada3419b9., state=RS_ZK_REGION_OPENING, servername=asf001.sp2.ygridcore.net,49518,1365336962323, createTime=1365336963197, payload.length=0 2013-04-07 12:16:03,225 DEBUG [hbase-am-zkevent-worker-pool-2-thread-26] master.AssignmentManager(740): Handling transition=RS_ZK_REGION_OPENING, server=asf001.sp2.ygridcore.net,49518,1365336962323, region=38f0eb146fb473b19541deaada3419b9, current state from region state map =null 2013-04-07 12:16:03,226 DEBUG [hbase-am-zkevent-worker-pool-2-thread-26] zookeeper.ZKUtil(1680): hconnection-0x1f327e-0x13de46b5f8b0003 Retrieved 43 byte(s) of data from znode /hbase/meta-region-server; data=asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:16:03,227 DEBUG [hbase-am-zkevent-worker-pool-2-thread-26] zookeeper.ZKUtil(1680): hconnection-0x1f327e-0x13de46b5f8b0003 Retrieved 43 byte(s) of data from znode /hbase/meta-region-server; data=asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:16:03,228 DEBUG [hbase-am-zkevent-worker-pool-2-thread-21] zookeeper.ZKUtil(1680): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Retrieved 156 byte(s) of data from znode /hbase/region-in-transition/dbb82f7f432d6664327da0b8f15b7d76 and set watcher; region=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,e,1365336962398.dbb82f7f432d6664327da0b8f15b7d76., state=RS_ZK_REGION_OPENED, servername=asf001.sp2.ygridcore.net,60884,1365336878389, createTime=1365336963155, payload.length=0 2013-04-07 12:16:03,229 DEBUG [MASTER_OPEN_REGION-asf001.sp2.ygridcore.net,33796,1365336878036-3] zookeeper.ZKUtil(1680): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Retrieved 156 byte(s) of data from znode /hbase/region-in-transition/dbb82f7f432d6664327da0b8f15b7d76; data=region=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,e,1365336962398.dbb82f7f432d6664327da0b8f15b7d76., state=RS_ZK_REGION_OPENED, servername=asf001.sp2.ygridcore.net,60884,1365336878389, createTime=1365336963155, payload.length=0 2013-04-07 12:16:03,229 DEBUG [Thread-200-EventThread] zookeeper.ZooKeeperWatcher(274): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/region-in-transition/eff8376859f5e6ba3201ec0ec9536d0f 2013-04-07 12:16:03,229 INFO [hbase-am-zkevent-worker-pool-2-thread-26] master.RegionStates(264): Region {NAME => 'testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,j,1365336962398.38f0eb146fb473b19541deaada3419b9.', STARTKEY => 'j', ENDKEY => '', ENCODED => 38f0eb146fb473b19541deaada3419b9,} transitioned from {testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,j,1365336962398.38f0eb146fb473b19541deaada3419b9. state=OFFLINE, ts=1365336963229, server=null} to {testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,j,1365336962398.38f0eb146fb473b19541deaada3419b9. state=OPENING, ts=1365336963229, server=asf001.sp2.ygridcore.net,49518,1365336962323} 2013-04-07 12:16:03,229 DEBUG [Thread-200-EventThread] zookeeper.ZooKeeperWatcher(274): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/region-in-transition 2013-04-07 12:16:03,229 DEBUG [hbase-am-zkevent-worker-pool-2-thread-35] master.AssignmentManager$4(1159): The znode of region testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,f,1365336962398.eff8376859f5e6ba3201ec0ec9536d0f. has been deleted. 2013-04-07 12:16:03,229 INFO [hbase-am-zkevent-worker-pool-2-thread-35] master.RegionStates(264): Region {NAME => 'testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,f,1365336962398.eff8376859f5e6ba3201ec0ec9536d0f.', STARTKEY => 'f', ENDKEY => 'g', ENCODED => eff8376859f5e6ba3201ec0ec9536d0f,} transitioned from {testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,f,1365336962398.eff8376859f5e6ba3201ec0ec9536d0f. state=OPEN, ts=1365336963187, server=asf001.sp2.ygridcore.net,60884,1365336878389} to {testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,f,1365336962398.eff8376859f5e6ba3201ec0ec9536d0f. state=OPEN, ts=1365336963229, server=asf001.sp2.ygridcore.net,60884,1365336878389} 2013-04-07 12:16:03,230 DEBUG [hbase-am-zkevent-worker-pool-20-thread-36] master.AssignmentManager$4(1159): The znode of region testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,f,1365336962398.eff8376859f5e6ba3201ec0ec9536d0f. has been deleted. 2013-04-07 12:16:03,230 INFO [hbase-am-zkevent-worker-pool-2-thread-35] master.RegionStates(299): Onlined region {NAME => 'testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,f,1365336962398.eff8376859f5e6ba3201ec0ec9536d0f.', STARTKEY => 'f', ENDKEY => 'g', ENCODED => eff8376859f5e6ba3201ec0ec9536d0f,} on asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:16:03,230 INFO [hbase-am-zkevent-worker-pool-20-thread-36] master.RegionStates(264): Region {NAME => 'testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,f,1365336962398.eff8376859f5e6ba3201ec0ec9536d0f.', STARTKEY => 'f', ENDKEY => 'g', ENCODED => eff8376859f5e6ba3201ec0ec9536d0f,} transitioned from {testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,f,1365336962398.eff8376859f5e6ba3201ec0ec9536d0f. state=OPEN, ts=1365336963179, server=asf001.sp2.ygridcore.net,60884,1365336878389} to {testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,f,1365336962398.eff8376859f5e6ba3201ec0ec9536d0f. state=OPEN, ts=1365336963230, server=asf001.sp2.ygridcore.net,60884,1365336878389} 2013-04-07 12:16:03,230 INFO [hbase-am-zkevent-worker-pool-2-thread-35] master.AssignmentManager$4(1164): The master has opened the region testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,f,1365336962398.eff8376859f5e6ba3201ec0ec9536d0f. that was online on asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:16:03,230 INFO [hbase-am-zkevent-worker-pool-20-thread-36] master.RegionStates(299): Onlined region {NAME => 'testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,f,1365336962398.eff8376859f5e6ba3201ec0ec9536d0f.', STARTKEY => 'f', ENDKEY => 'g', ENCODED => eff8376859f5e6ba3201ec0ec9536d0f,} on asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:16:03,230 DEBUG [hbase-am-zkevent-worker-pool-2-thread-21] master.AssignmentManager(740): Handling transition=RS_ZK_REGION_OPENED, server=asf001.sp2.ygridcore.net,60884,1365336878389, region=dbb82f7f432d6664327da0b8f15b7d76, current state from region state map ={testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,e,1365336962398.dbb82f7f432d6664327da0b8f15b7d76. state=OPEN, ts=1365336963196, server=asf001.sp2.ygridcore.net,60884,1365336878389} 2013-04-07 12:16:03,230 WARN [hbase-am-zkevent-worker-pool-2-thread-21] master.AssignmentManager(934): Received OPENED for region dbb82f7f432d6664327da0b8f15b7d76 from server asf001.sp2.ygridcore.net,60884,1365336878389 but region was in the state {testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,e,1365336962398.dbb82f7f432d6664327da0b8f15b7d76. state=OPEN, ts=1365336963196, server=asf001.sp2.ygridcore.net,60884,1365336878389} and not in expected PENDING_OPEN or OPENING states, or not on the expected server 2013-04-07 12:16:03,230 INFO [hbase-am-zkevent-worker-pool-20-thread-36] master.AssignmentManager$4(1164): The master has opened the region testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,f,1365336962398.eff8376859f5e6ba3201ec0ec9536d0f. that was online on asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:16:03,231 DEBUG [MASTER_OPEN_REGION-asf001.sp2.ygridcore.net,33796,1365336878036-1] zookeeper.ZKAssign(434): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Successfully deleted unassigned node for region eff8376859f5e6ba3201ec0ec9536d0f in expected state RS_ZK_REGION_OPENED 2013-04-07 12:16:03,231 INFO [PRI IPC Server handler 8 on 60884] regionserver.HRegionServer(3434): Received close region: dbb82f7f432d6664327da0b8f15b7d76Transitioning in ZK: no. Version of ZK closing node:-1. Destination server:null 2013-04-07 12:16:03,232 DEBUG [MASTER_OPEN_REGION-asf001.sp2.ygridcore.net,33796,1365336878036-1] handler.OpenedRegionHandler(146): Handling OPENED event for testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,g,1365336962398.682475e1358b7306bd3406e9330263a4. from asf001.sp2.ygridcore.net,49518,1365336962323; deleting unassigned node 2013-04-07 12:16:03,232 DEBUG [MASTER_OPEN_REGION-asf001.sp2.ygridcore.net,33796,1365336878036-1] zookeeper.ZKAssign(405): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Deleting existing unassigned node for 682475e1358b7306bd3406e9330263a4 that is in expected state RS_ZK_REGION_OPENED 2013-04-07 12:16:03,232 DEBUG [RS_CLOSE_REGION-asf001.sp2.ygridcore.net,60884,1365336878389-2] handler.CloseRegionHandler(125): Processing close of testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,e,1365336962398.dbb82f7f432d6664327da0b8f15b7d76. 2013-04-07 12:16:03,232 DEBUG [hbase-am-zkevent-worker-pool-2-thread-21] master.AssignmentManager(1638): Sent CLOSE to asf001.sp2.ygridcore.net,60884,1365336878389 for region testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,e,1365336962398.dbb82f7f432d6664327da0b8f15b7d76. 2013-04-07 12:16:03,232 DEBUG [RS_CLOSE_REGION-asf001.sp2.ygridcore.net,60884,1365336878389-2] regionserver.HRegion(889): Closing testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,e,1365336962398.dbb82f7f432d6664327da0b8f15b7d76.: disabling compactions & flushes 2013-04-07 12:16:03,233 DEBUG [RS_CLOSE_REGION-asf001.sp2.ygridcore.net,60884,1365336878389-2] regionserver.HRegion(911): Updates disabled for region testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,e,1365336962398.dbb82f7f432d6664327da0b8f15b7d76. 2013-04-07 12:16:03,233 INFO [RS_CLOSE_REGION-asf001.sp2.ygridcore.net,60884,1365336878389-2] regionserver.HRegion(968): Closed testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,e,1365336962398.dbb82f7f432d6664327da0b8f15b7d76. 2013-04-07 12:16:03,233 DEBUG [RS_CLOSE_REGION-asf001.sp2.ygridcore.net,60884,1365336878389-2] handler.CloseRegionHandler(177): Closed region testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,e,1365336962398.dbb82f7f432d6664327da0b8f15b7d76. 2013-04-07 12:16:03,233 DEBUG [hbase-am-zkevent-worker-pool-2-thread-32] zookeeper.ZKUtil(1680): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Retrieved 156 byte(s) of data from znode /hbase/region-in-transition/b1b340dff1e06fd6cb0b0a9c64964551 and set watcher; region=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,a,1365336962398.b1b340dff1e06fd6cb0b0a9c64964551., state=RS_ZK_REGION_OPENED, servername=asf001.sp2.ygridcore.net,49518,1365336962323, createTime=1365336963169, payload.length=0 2013-04-07 12:16:03,233 DEBUG [hbase-am-zkevent-worker-pool-2-thread-32] master.AssignmentManager(740): Handling transition=RS_ZK_REGION_OPENED, server=asf001.sp2.ygridcore.net,49518,1365336962323, region=b1b340dff1e06fd6cb0b0a9c64964551, current state from region state map ={testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,a,1365336962398.b1b340dff1e06fd6cb0b0a9c64964551. state=OPEN, ts=1365336963201, server=asf001.sp2.ygridcore.net,49518,1365336962323} 2013-04-07 12:16:03,233 WARN [hbase-am-zkevent-worker-pool-2-thread-32] master.AssignmentManager(934): Received OPENED for region b1b340dff1e06fd6cb0b0a9c64964551 from server asf001.sp2.ygridcore.net,49518,1365336962323 but region was in the state {testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,a,1365336962398.b1b340dff1e06fd6cb0b0a9c64964551. state=OPEN, ts=1365336963201, server=asf001.sp2.ygridcore.net,49518,1365336962323} and not in expected PENDING_OPEN or OPENING states, or not on the expected server 2013-04-07 12:16:03,234 DEBUG [Thread-200-EventThread] zookeeper.ZooKeeperWatcher(274): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/region-in-transition/dbb82f7f432d6664327da0b8f15b7d76 2013-04-07 12:16:03,234 DEBUG [hbase-am-zkevent-worker-pool-20-thread-39] master.AssignmentManager$4(1159): The znode of region testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,e,1365336962398.dbb82f7f432d6664327da0b8f15b7d76. has been deleted. 2013-04-07 12:16:03,234 INFO [hbase-am-zkevent-worker-pool-20-thread-39] master.RegionStates(264): Region {NAME => 'testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,e,1365336962398.dbb82f7f432d6664327da0b8f15b7d76.', STARTKEY => 'e', ENDKEY => 'f', ENCODED => dbb82f7f432d6664327da0b8f15b7d76,} transitioned from {testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,e,1365336962398.dbb82f7f432d6664327da0b8f15b7d76. state=OPEN, ts=1365336963181, server=asf001.sp2.ygridcore.net,60884,1365336878389} to {testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,e,1365336962398.dbb82f7f432d6664327da0b8f15b7d76. state=OPEN, ts=1365336963234, server=asf001.sp2.ygridcore.net,60884,1365336878389} 2013-04-07 12:16:03,234 DEBUG [hbase-am-zkevent-worker-pool-2-thread-23] master.AssignmentManager$4(1159): The znode of region testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,e,1365336962398.dbb82f7f432d6664327da0b8f15b7d76. has been deleted. 2013-04-07 12:16:03,235 INFO [hbase-am-zkevent-worker-pool-20-thread-39] master.RegionStates(299): Onlined region {NAME => 'testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,e,1365336962398.dbb82f7f432d6664327da0b8f15b7d76.', STARTKEY => 'e', ENDKEY => 'f', ENCODED => dbb82f7f432d6664327da0b8f15b7d76,} on asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:16:03,235 INFO [hbase-am-zkevent-worker-pool-2-thread-23] master.RegionStates(264): Region {NAME => 'testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,e,1365336962398.dbb82f7f432d6664327da0b8f15b7d76.', STARTKEY => 'e', ENDKEY => 'f', ENCODED => dbb82f7f432d6664327da0b8f15b7d76,} transitioned from {testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,e,1365336962398.dbb82f7f432d6664327da0b8f15b7d76. state=OPEN, ts=1365336963196, server=asf001.sp2.ygridcore.net,60884,1365336878389} to {testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,e,1365336962398.dbb82f7f432d6664327da0b8f15b7d76. state=OPEN, ts=1365336963235, server=asf001.sp2.ygridcore.net,60884,1365336878389} 2013-04-07 12:16:03,235 INFO [hbase-am-zkevent-worker-pool-20-thread-39] master.AssignmentManager$4(1164): The master has opened the region testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,e,1365336962398.dbb82f7f432d6664327da0b8f15b7d76. that was online on asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:16:03,235 DEBUG [MASTER_OPEN_REGION-asf001.sp2.ygridcore.net,33796,1365336878036-4] zookeeper.ZKAssign(434): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Successfully deleted unassigned node for region dbb82f7f432d6664327da0b8f15b7d76 in expected state RS_ZK_REGION_OPENED 2013-04-07 12:16:03,236 DEBUG [MASTER_OPEN_REGION-asf001.sp2.ygridcore.net,33796,1365336878036-4] handler.OpenedRegionHandler(146): Handling OPENED event for testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,d,1365336962398.e99798d2c4d566ddb063efe03842457f. from asf001.sp2.ygridcore.net,49518,1365336962323; deleting unassigned node 2013-04-07 12:16:03,236 DEBUG [MASTER_OPEN_REGION-asf001.sp2.ygridcore.net,33796,1365336878036-4] zookeeper.ZKAssign(405): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Deleting existing unassigned node for e99798d2c4d566ddb063efe03842457f that is in expected state RS_ZK_REGION_OPENED 2013-04-07 12:16:03,235 INFO [hbase-am-zkevent-worker-pool-2-thread-23] master.RegionStates(299): Onlined region {NAME => 'testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,e,1365336962398.dbb82f7f432d6664327da0b8f15b7d76.', STARTKEY => 'e', ENDKEY => 'f', ENCODED => dbb82f7f432d6664327da0b8f15b7d76,} on asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:16:03,235 DEBUG [hbase-am-zkevent-worker-pool-2-thread-32] master.AssignmentManager(1638): Sent CLOSE to asf001.sp2.ygridcore.net,49518,1365336962323 for region testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,a,1365336962398.b1b340dff1e06fd6cb0b0a9c64964551. 2013-04-07 12:16:03,236 INFO [hbase-am-zkevent-worker-pool-2-thread-23] master.AssignmentManager$4(1164): The master has opened the region testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,e,1365336962398.dbb82f7f432d6664327da0b8f15b7d76. that was online on asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:16:03,236 DEBUG [MASTER_OPEN_REGION-asf001.sp2.ygridcore.net,33796,1365336878036-0] zookeeper.ZKUtil(1680): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Retrieved 156 byte(s) of data from znode /hbase/region-in-transition/b1b340dff1e06fd6cb0b0a9c64964551; data=region=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,a,1365336962398.b1b340dff1e06fd6cb0b0a9c64964551., state=RS_ZK_REGION_OPENED, servername=asf001.sp2.ygridcore.net,49518,1365336962323, createTime=1365336963169, payload.length=0 2013-04-07 12:16:03,237 DEBUG [Thread-200-EventThread] zookeeper.ZooKeeperWatcher(274): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/region-in-transition/11c8123e8d563f264d947ffa106e4c7c 2013-04-07 12:16:03,238 DEBUG [Thread-200-EventThread] zookeeper.ZooKeeperWatcher(274): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/region-in-transition/38f0eb146fb473b19541deaada3419b9 2013-04-07 12:16:03,239 WARN [MASTER_OPEN_REGION-asf001.sp2.ygridcore.net,33796,1365336878036-2] zookeeper.RecoverableZooKeeper(161): Node /hbase/region-in-transition/eff8376859f5e6ba3201ec0ec9536d0f already deleted, retry=false 2013-04-07 12:16:03,239 WARN [MASTER_OPEN_REGION-asf001.sp2.ygridcore.net,33796,1365336878036-2] handler.OpenedRegionHandler(132): The znode of the region testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,f,1365336962398.eff8376859f5e6ba3201ec0ec9536d0f. would have already been deleted 2013-04-07 12:16:03,239 ERROR [MASTER_OPEN_REGION-asf001.sp2.ygridcore.net,33796,1365336878036-2] handler.OpenedRegionHandler(104): The znode of region testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,f,1365336962398.eff8376859f5e6ba3201ec0ec9536d0f. could not be deleted. 2013-04-07 12:16:03,239 DEBUG [MASTER_OPEN_REGION-asf001.sp2.ygridcore.net,33796,1365336878036-2] handler.OpenedRegionHandler(146): Handling OPENED event for testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,d,1365336962398.e99798d2c4d566ddb063efe03842457f. from asf001.sp2.ygridcore.net,49518,1365336962323; deleting unassigned node 2013-04-07 12:16:03,239 DEBUG [MASTER_OPEN_REGION-asf001.sp2.ygridcore.net,33796,1365336878036-2] zookeeper.ZKAssign(405): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Deleting existing unassigned node for e99798d2c4d566ddb063efe03842457f that is in expected state RS_ZK_REGION_OPENED 2013-04-07 12:16:03,240 DEBUG [hbase-am-zkevent-worker-pool-2-thread-30] zookeeper.ZKUtil(1680): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Retrieved 156 byte(s) of data from znode /hbase/region-in-transition/38f0eb146fb473b19541deaada3419b9 and set watcher; region=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,j,1365336962398.38f0eb146fb473b19541deaada3419b9., state=RS_ZK_REGION_OPENED, servername=asf001.sp2.ygridcore.net,49518,1365336962323, createTime=1365336963218, payload.length=0 2013-04-07 12:16:03,241 WARN [MASTER_OPEN_REGION-asf001.sp2.ygridcore.net,33796,1365336878036-3] zookeeper.RecoverableZooKeeper(161): Node /hbase/region-in-transition/dbb82f7f432d6664327da0b8f15b7d76 already deleted, retry=false 2013-04-07 12:16:03,241 WARN [MASTER_OPEN_REGION-asf001.sp2.ygridcore.net,33796,1365336878036-3] handler.OpenedRegionHandler(132): The znode of the region testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,e,1365336962398.dbb82f7f432d6664327da0b8f15b7d76. would have already been deleted 2013-04-07 12:16:03,241 ERROR [MASTER_OPEN_REGION-asf001.sp2.ygridcore.net,33796,1365336878036-3] handler.OpenedRegionHandler(104): The znode of region testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,e,1365336962398.dbb82f7f432d6664327da0b8f15b7d76. could not be deleted. 2013-04-07 12:16:03,241 DEBUG [MASTER_OPEN_REGION-asf001.sp2.ygridcore.net,33796,1365336878036-3] handler.OpenedRegionHandler(146): Handling OPENED event for testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,g,1365336962398.682475e1358b7306bd3406e9330263a4. from asf001.sp2.ygridcore.net,49518,1365336962323; deleting unassigned node 2013-04-07 12:16:03,241 DEBUG [MASTER_OPEN_REGION-asf001.sp2.ygridcore.net,33796,1365336878036-3] zookeeper.ZKAssign(405): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Deleting existing unassigned node for 682475e1358b7306bd3406e9330263a4 that is in expected state RS_ZK_REGION_OPENED 2013-04-07 12:16:03,242 DEBUG [hbase-am-zkevent-worker-pool-20-thread-35] zookeeper.ZKUtil(1680): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Retrieved 156 byte(s) of data from znode /hbase/region-in-transition/38f0eb146fb473b19541deaada3419b9 and set watcher; region=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,j,1365336962398.38f0eb146fb473b19541deaada3419b9., state=RS_ZK_REGION_OPENED, servername=asf001.sp2.ygridcore.net,49518,1365336962323, createTime=1365336963218, payload.length=0 2013-04-07 12:16:03,242 DEBUG [hbase-am-zkevent-worker-pool-20-thread-35] master.AssignmentManager(740): Handling transition=RS_ZK_REGION_OPENED, server=asf001.sp2.ygridcore.net,49518,1365336962323, region=38f0eb146fb473b19541deaada3419b9, current state from region state map ={testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,j,1365336962398.38f0eb146fb473b19541deaada3419b9. state=OPENING, ts=1365336963222, server=asf001.sp2.ygridcore.net,49518,1365336962323} 2013-04-07 12:16:03,242 INFO [hbase-am-zkevent-worker-pool-20-thread-35] master.RegionStates(264): Region {NAME => 'testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,j,1365336962398.38f0eb146fb473b19541deaada3419b9.', STARTKEY => 'j', ENDKEY => '', ENCODED => 38f0eb146fb473b19541deaada3419b9,} transitioned from {testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,j,1365336962398.38f0eb146fb473b19541deaada3419b9. state=OPENING, ts=1365336963222, server=asf001.sp2.ygridcore.net,49518,1365336962323} to {testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,j,1365336962398.38f0eb146fb473b19541deaada3419b9. state=OPEN, ts=1365336963242, server=asf001.sp2.ygridcore.net,49518,1365336962323} 2013-04-07 12:16:03,242 DEBUG [hbase-am-zkevent-worker-pool-2-thread-26] zookeeper.ZKUtil(1680): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Retrieved 156 byte(s) of data from znode /hbase/region-in-transition/38f0eb146fb473b19541deaada3419b9 and set watcher; region=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,j,1365336962398.38f0eb146fb473b19541deaada3419b9., state=RS_ZK_REGION_OPENED, servername=asf001.sp2.ygridcore.net,49518,1365336962323, createTime=1365336963218, payload.length=0 2013-04-07 12:16:03,242 DEBUG [hbase-am-zkevent-worker-pool-2-thread-26] master.AssignmentManager(740): Handling transition=RS_ZK_REGION_OPENED, server=asf001.sp2.ygridcore.net,49518,1365336962323, region=38f0eb146fb473b19541deaada3419b9, current state from region state map ={testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,j,1365336962398.38f0eb146fb473b19541deaada3419b9. state=OPENING, ts=1365336963229, server=asf001.sp2.ygridcore.net,49518,1365336962323} 2013-04-07 12:16:03,242 INFO [hbase-am-zkevent-worker-pool-2-thread-26] master.RegionStates(264): Region {NAME => 'testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,j,1365336962398.38f0eb146fb473b19541deaada3419b9.', STARTKEY => 'j', ENDKEY => '', ENCODED => 38f0eb146fb473b19541deaada3419b9,} transitioned from {testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,j,1365336962398.38f0eb146fb473b19541deaada3419b9. state=OPENING, ts=1365336963229, server=asf001.sp2.ygridcore.net,49518,1365336962323} to {testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,j,1365336962398.38f0eb146fb473b19541deaada3419b9. state=OPEN, ts=1365336963242, server=asf001.sp2.ygridcore.net,49518,1365336962323} 2013-04-07 12:16:03,243 DEBUG [MASTER_OPEN_REGION-asf001.sp2.ygridcore.net,33796,1365336878036-1] zookeeper.ZKUtil(1680): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Retrieved 156 byte(s) of data from znode /hbase/region-in-transition/682475e1358b7306bd3406e9330263a4; data=region=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,g,1365336962398.682475e1358b7306bd3406e9330263a4., state=RS_ZK_REGION_OPENED, servername=asf001.sp2.ygridcore.net,49518,1365336962323, createTime=1365336963154, payload.length=0 2013-04-07 12:16:03,244 DEBUG [MASTER_OPEN_REGION-asf001.sp2.ygridcore.net,33796,1365336878036-4] zookeeper.ZKUtil(1680): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Retrieved 156 byte(s) of data from znode /hbase/region-in-transition/e99798d2c4d566ddb063efe03842457f; data=region=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,d,1365336962398.e99798d2c4d566ddb063efe03842457f., state=RS_ZK_REGION_OPENED, servername=asf001.sp2.ygridcore.net,49518,1365336962323, createTime=1365336963163, payload.length=0 2013-04-07 12:16:03,245 DEBUG [Thread-200-EventThread] zookeeper.ZooKeeperWatcher(274): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/region-in-transition/b1b340dff1e06fd6cb0b0a9c64964551 2013-04-07 12:16:03,245 DEBUG [hbase-am-zkevent-worker-pool-20-thread-40] master.AssignmentManager$4(1159): The znode of region testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,a,1365336962398.b1b340dff1e06fd6cb0b0a9c64964551. has been deleted. 2013-04-07 12:16:03,245 INFO [hbase-am-zkevent-worker-pool-20-thread-40] master.RegionStates(264): Region {NAME => 'testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,a,1365336962398.b1b340dff1e06fd6cb0b0a9c64964551.', STARTKEY => 'a', ENDKEY => 'b', ENCODED => b1b340dff1e06fd6cb0b0a9c64964551,} transitioned from {testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,a,1365336962398.b1b340dff1e06fd6cb0b0a9c64964551. state=OPEN, ts=1365336963218, server=asf001.sp2.ygridcore.net,49518,1365336962323} to {testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,a,1365336962398.b1b340dff1e06fd6cb0b0a9c64964551. state=OPEN, ts=1365336963245, server=asf001.sp2.ygridcore.net,49518,1365336962323} 2013-04-07 12:16:03,245 DEBUG [Thread-200-EventThread] zookeeper.ZooKeeperWatcher(274): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/region-in-transition 2013-04-07 12:16:03,245 DEBUG [hbase-am-zkevent-worker-pool-2-thread-40] master.AssignmentManager$4(1159): The znode of region testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,a,1365336962398.b1b340dff1e06fd6cb0b0a9c64964551. has been deleted. 2013-04-07 12:16:03,246 INFO [hbase-am-zkevent-worker-pool-2-thread-40] master.RegionStates(264): Region {NAME => 'testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,a,1365336962398.b1b340dff1e06fd6cb0b0a9c64964551.', STARTKEY => 'a', ENDKEY => 'b', ENCODED => b1b340dff1e06fd6cb0b0a9c64964551,} transitioned from {testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,a,1365336962398.b1b340dff1e06fd6cb0b0a9c64964551. state=OPEN, ts=1365336963201, server=asf001.sp2.ygridcore.net,49518,1365336962323} to {testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,a,1365336962398.b1b340dff1e06fd6cb0b0a9c64964551. state=OPEN, ts=1365336963246, server=asf001.sp2.ygridcore.net,49518,1365336962323} 2013-04-07 12:16:03,246 INFO [hbase-am-zkevent-worker-pool-2-thread-40] master.RegionStates(299): Onlined region {NAME => 'testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,a,1365336962398.b1b340dff1e06fd6cb0b0a9c64964551.', STARTKEY => 'a', ENDKEY => 'b', ENCODED => b1b340dff1e06fd6cb0b0a9c64964551,} on asf001.sp2.ygridcore.net,49518,1365336962323 2013-04-07 12:16:03,246 INFO [hbase-am-zkevent-worker-pool-2-thread-40] master.AssignmentManager$4(1164): The master has opened the region testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,a,1365336962398.b1b340dff1e06fd6cb0b0a9c64964551. that was online on asf001.sp2.ygridcore.net,49518,1365336962323 2013-04-07 12:16:03,246 DEBUG [MASTER_OPEN_REGION-asf001.sp2.ygridcore.net,33796,1365336878036-0] zookeeper.ZKAssign(434): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Successfully deleted unassigned node for region b1b340dff1e06fd6cb0b0a9c64964551 in expected state RS_ZK_REGION_OPENED 2013-04-07 12:16:03,245 INFO [hbase-am-zkevent-worker-pool-20-thread-40] master.RegionStates(299): Onlined region {NAME => 'testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,a,1365336962398.b1b340dff1e06fd6cb0b0a9c64964551.', STARTKEY => 'a', ENDKEY => 'b', ENCODED => b1b340dff1e06fd6cb0b0a9c64964551,} on asf001.sp2.ygridcore.net,49518,1365336962323 2013-04-07 12:16:03,246 DEBUG [MASTER_OPEN_REGION-asf001.sp2.ygridcore.net,33796,1365336878036-0] handler.OpenedRegionHandler(146): Handling OPENED event for testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,h,1365336962398.1ce6d62427641e2a0e135d3d27b54b5d. from asf001.sp2.ygridcore.net,43802,1365336889418; deleting unassigned node 2013-04-07 12:16:03,246 INFO [hbase-am-zkevent-worker-pool-20-thread-40] master.AssignmentManager$4(1164): The master has opened the region testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,a,1365336962398.b1b340dff1e06fd6cb0b0a9c64964551. that was online on asf001.sp2.ygridcore.net,49518,1365336962323 2013-04-07 12:16:03,246 DEBUG [MASTER_OPEN_REGION-asf001.sp2.ygridcore.net,33796,1365336878036-0] zookeeper.ZKAssign(405): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Deleting existing unassigned node for 1ce6d62427641e2a0e135d3d27b54b5d that is in expected state RS_ZK_REGION_OPENED 2013-04-07 12:16:03,247 DEBUG [hbase-am-zkevent-worker-pool-20-thread-38] zookeeper.ZKUtil(1680): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Retrieved 156 byte(s) of data from znode /hbase/region-in-transition/11c8123e8d563f264d947ffa106e4c7c and set watcher; region=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,b,1365336962398.11c8123e8d563f264d947ffa106e4c7c., state=RS_ZK_REGION_OPENED, servername=asf001.sp2.ygridcore.net,60884,1365336878389, createTime=1365336963214, payload.length=0 2013-04-07 12:16:03,247 DEBUG [hbase-am-zkevent-worker-pool-20-thread-38] master.AssignmentManager(740): Handling transition=RS_ZK_REGION_OPENED, server=asf001.sp2.ygridcore.net,60884,1365336878389, region=11c8123e8d563f264d947ffa106e4c7c, current state from region state map ={testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,b,1365336962398.11c8123e8d563f264d947ffa106e4c7c. state=OPENING, ts=1365336963213, server=asf001.sp2.ygridcore.net,60884,1365336878389} 2013-04-07 12:16:03,247 INFO [hbase-am-zkevent-worker-pool-20-thread-38] master.RegionStates(264): Region {NAME => 'testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,b,1365336962398.11c8123e8d563f264d947ffa106e4c7c.', STARTKEY => 'b', ENDKEY => 'c', ENCODED => 11c8123e8d563f264d947ffa106e4c7c,} transitioned from {testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,b,1365336962398.11c8123e8d563f264d947ffa106e4c7c. state=OPENING, ts=1365336963213, server=asf001.sp2.ygridcore.net,60884,1365336878389} to {testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,b,1365336962398.11c8123e8d563f264d947ffa106e4c7c. state=OPEN, ts=1365336963247, server=asf001.sp2.ygridcore.net,60884,1365336878389} 2013-04-07 12:16:03,248 DEBUG [hbase-am-zkevent-worker-pool-2-thread-29] zookeeper.ZKUtil(1680): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Retrieved 156 byte(s) of data from znode /hbase/region-in-transition/11c8123e8d563f264d947ffa106e4c7c and set watcher; region=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,b,1365336962398.11c8123e8d563f264d947ffa106e4c7c., state=RS_ZK_REGION_OPENED, servername=asf001.sp2.ygridcore.net,60884,1365336878389, createTime=1365336963214, payload.length=0 2013-04-07 12:16:03,248 DEBUG [hbase-am-zkevent-worker-pool-2-thread-29] master.AssignmentManager(740): Handling transition=RS_ZK_REGION_OPENED, server=asf001.sp2.ygridcore.net,60884,1365336878389, region=11c8123e8d563f264d947ffa106e4c7c, current state from region state map ={testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,b,1365336962398.11c8123e8d563f264d947ffa106e4c7c. state=OPENING, ts=1365336963222, server=asf001.sp2.ygridcore.net,60884,1365336878389} 2013-04-07 12:16:03,249 INFO [hbase-am-zkevent-worker-pool-2-thread-29] master.RegionStates(264): Region {NAME => 'testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,b,1365336962398.11c8123e8d563f264d947ffa106e4c7c.', STARTKEY => 'b', ENDKEY => 'c', ENCODED => 11c8123e8d563f264d947ffa106e4c7c,} transitioned from {testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,b,1365336962398.11c8123e8d563f264d947ffa106e4c7c. state=OPENING, ts=1365336963222, server=asf001.sp2.ygridcore.net,60884,1365336878389} to {testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,b,1365336962398.11c8123e8d563f264d947ffa106e4c7c. state=OPEN, ts=1365336963249, server=asf001.sp2.ygridcore.net,60884,1365336878389} 2013-04-07 12:16:03,249 DEBUG [MASTER_OPEN_REGION-asf001.sp2.ygridcore.net,33796,1365336878036-2] zookeeper.ZKUtil(1680): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Retrieved 156 byte(s) of data from znode /hbase/region-in-transition/e99798d2c4d566ddb063efe03842457f; data=region=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,d,1365336962398.e99798d2c4d566ddb063efe03842457f., state=RS_ZK_REGION_OPENED, servername=asf001.sp2.ygridcore.net,49518,1365336962323, createTime=1365336963163, payload.length=0 2013-04-07 12:16:03,251 DEBUG [hbase-am-zkevent-worker-pool-20-thread-35] zookeeper.ZKUtil(1680): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Retrieved 156 byte(s) of data from znode /hbase/region-in-transition/38f0eb146fb473b19541deaada3419b9 and set watcher; region=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,j,1365336962398.38f0eb146fb473b19541deaada3419b9., state=RS_ZK_REGION_OPENED, servername=asf001.sp2.ygridcore.net,49518,1365336962323, createTime=1365336963218, payload.length=0 2013-04-07 12:16:03,251 DEBUG [hbase-am-zkevent-worker-pool-20-thread-35] master.AssignmentManager(740): Handling transition=RS_ZK_REGION_OPENED, server=asf001.sp2.ygridcore.net,49518,1365336962323, region=38f0eb146fb473b19541deaada3419b9, current state from region state map ={testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,j,1365336962398.38f0eb146fb473b19541deaada3419b9. state=OPEN, ts=1365336963242, server=asf001.sp2.ygridcore.net,49518,1365336962323} 2013-04-07 12:16:03,251 DEBUG [MASTER_OPEN_REGION-asf001.sp2.ygridcore.net,33796,1365336878036-3] zookeeper.ZKUtil(1680): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Retrieved 156 byte(s) of data from znode /hbase/region-in-transition/682475e1358b7306bd3406e9330263a4; data=region=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,g,1365336962398.682475e1358b7306bd3406e9330263a4., state=RS_ZK_REGION_OPENED, servername=asf001.sp2.ygridcore.net,49518,1365336962323, createTime=1365336963154, payload.length=0 2013-04-07 12:16:03,251 WARN [hbase-am-zkevent-worker-pool-20-thread-35] master.AssignmentManager(934): Received OPENED for region 38f0eb146fb473b19541deaada3419b9 from server asf001.sp2.ygridcore.net,49518,1365336962323 but region was in the state {testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,j,1365336962398.38f0eb146fb473b19541deaada3419b9. state=OPEN, ts=1365336963242, server=asf001.sp2.ygridcore.net,49518,1365336962323} and not in expected PENDING_OPEN or OPENING states, or not on the expected server 2013-04-07 12:16:03,252 DEBUG [hbase-am-zkevent-worker-pool-2-thread-26] zookeeper.ZKUtil(1680): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Retrieved 156 byte(s) of data from znode /hbase/region-in-transition/38f0eb146fb473b19541deaada3419b9 and set watcher; region=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,j,1365336962398.38f0eb146fb473b19541deaada3419b9., state=RS_ZK_REGION_OPENED, servername=asf001.sp2.ygridcore.net,49518,1365336962323, createTime=1365336963218, payload.length=0 2013-04-07 12:16:03,252 DEBUG [hbase-am-zkevent-worker-pool-2-thread-26] master.AssignmentManager(740): Handling transition=RS_ZK_REGION_OPENED, server=asf001.sp2.ygridcore.net,49518,1365336962323, region=38f0eb146fb473b19541deaada3419b9, current state from region state map ={testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,j,1365336962398.38f0eb146fb473b19541deaada3419b9. state=OPEN, ts=1365336963242, server=asf001.sp2.ygridcore.net,49518,1365336962323} 2013-04-07 12:16:03,254 WARN [hbase-am-zkevent-worker-pool-2-thread-26] master.AssignmentManager(934): Received OPENED for region 38f0eb146fb473b19541deaada3419b9 from server asf001.sp2.ygridcore.net,49518,1365336962323 but region was in the state {testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,j,1365336962398.38f0eb146fb473b19541deaada3419b9. state=OPEN, ts=1365336963242, server=asf001.sp2.ygridcore.net,49518,1365336962323} and not in expected PENDING_OPEN or OPENING states, or not on the expected server 2013-04-07 12:16:03,255 DEBUG [hbase-am-zkevent-worker-pool-20-thread-35] master.AssignmentManager(1638): Sent CLOSE to asf001.sp2.ygridcore.net,49518,1365336962323 for region testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,j,1365336962398.38f0eb146fb473b19541deaada3419b9. 2013-04-07 12:16:03,255 DEBUG [Thread-200-EventThread] zookeeper.ZooKeeperWatcher(274): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/region-in-transition/682475e1358b7306bd3406e9330263a4 2013-04-07 12:16:03,255 DEBUG [MASTER_OPEN_REGION-asf001.sp2.ygridcore.net,33796,1365336878036-1] zookeeper.ZKAssign(434): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Successfully deleted unassigned node for region 682475e1358b7306bd3406e9330263a4 in expected state RS_ZK_REGION_OPENED 2013-04-07 12:16:03,255 DEBUG [Thread-200-EventThread] zookeeper.ZooKeeperWatcher(274): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/region-in-transition/e99798d2c4d566ddb063efe03842457f 2013-04-07 12:16:03,256 DEBUG [MASTER_OPEN_REGION-asf001.sp2.ygridcore.net,33796,1365336878036-1] handler.OpenedRegionHandler(146): Handling OPENED event for testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,h,1365336962398.1ce6d62427641e2a0e135d3d27b54b5d. from asf001.sp2.ygridcore.net,43802,1365336889418; deleting unassigned node 2013-04-07 12:16:03,256 DEBUG [MASTER_OPEN_REGION-asf001.sp2.ygridcore.net,33796,1365336878036-4] zookeeper.ZKAssign(434): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Successfully deleted unassigned node for region e99798d2c4d566ddb063efe03842457f in expected state RS_ZK_REGION_OPENED 2013-04-07 12:16:03,256 DEBUG [hbase-am-zkevent-worker-pool-2-thread-33] master.AssignmentManager$4(1159): The znode of region testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,d,1365336962398.e99798d2c4d566ddb063efe03842457f. has been deleted. 2013-04-07 12:16:03,256 WARN [MASTER_OPEN_REGION-asf001.sp2.ygridcore.net,33796,1365336878036-4] handler.OpenedRegionHandler(108): Skipping the onlining of testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,a,1365336962398.b1b340dff1e06fd6cb0b0a9c64964551. because regions is NOT in RIT -- presuming this is because it SPLIT 2013-04-07 12:16:03,256 INFO [hbase-am-zkevent-worker-pool-2-thread-33] master.RegionStates(264): Region {NAME => 'testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,d,1365336962398.e99798d2c4d566ddb063efe03842457f.', STARTKEY => 'd', ENDKEY => 'e', ENCODED => e99798d2c4d566ddb063efe03842457f,} transitioned from {testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,d,1365336962398.e99798d2c4d566ddb063efe03842457f. state=OPEN, ts=1365336963204, server=asf001.sp2.ygridcore.net,49518,1365336962323} to {testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,d,1365336962398.e99798d2c4d566ddb063efe03842457f. state=OPEN, ts=1365336963256, server=asf001.sp2.ygridcore.net,49518,1365336962323} 2013-04-07 12:16:03,256 DEBUG [MASTER_OPEN_REGION-asf001.sp2.ygridcore.net,33796,1365336878036-4] handler.OpenedRegionHandler(146): Handling OPENED event for testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,i,1365336962398.385e5846630b0cb4493758961a2c2457. from asf001.sp2.ygridcore.net,60884,1365336878389; deleting unassigned node 2013-04-07 12:16:03,256 INFO [hbase-am-zkevent-worker-pool-2-thread-33] master.RegionStates(299): Onlined region {NAME => 'testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,d,1365336962398.e99798d2c4d566ddb063efe03842457f.', STARTKEY => 'd', ENDKEY => 'e', ENCODED => e99798d2c4d566ddb063efe03842457f,} on asf001.sp2.ygridcore.net,49518,1365336962323 2013-04-07 12:16:03,256 DEBUG [MASTER_OPEN_REGION-asf001.sp2.ygridcore.net,33796,1365336878036-4] zookeeper.ZKAssign(405): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Deleting existing unassigned node for 385e5846630b0cb4493758961a2c2457 that is in expected state RS_ZK_REGION_OPENED 2013-04-07 12:16:03,256 INFO [hbase-am-zkevent-worker-pool-2-thread-33] master.AssignmentManager$4(1164): The master has opened the region testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,d,1365336962398.e99798d2c4d566ddb063efe03842457f. that was online on asf001.sp2.ygridcore.net,49518,1365336962323 2013-04-07 12:16:03,255 DEBUG [hbase-am-zkevent-worker-pool-2-thread-24] master.AssignmentManager$4(1159): The znode of region testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,g,1365336962398.682475e1358b7306bd3406e9330263a4. has been deleted. 2013-04-07 12:16:03,256 DEBUG [MASTER_OPEN_REGION-asf001.sp2.ygridcore.net,33796,1365336878036-1] zookeeper.ZKAssign(405): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Deleting existing unassigned node for 1ce6d62427641e2a0e135d3d27b54b5d that is in expected state RS_ZK_REGION_OPENED 2013-04-07 12:16:03,257 INFO [hbase-am-zkevent-worker-pool-2-thread-24] master.RegionStates(264): Region {NAME => 'testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,g,1365336962398.682475e1358b7306bd3406e9330263a4.', STARTKEY => 'g', ENDKEY => 'h', ENCODED => 682475e1358b7306bd3406e9330263a4,} transitioned from {testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,g,1365336962398.682475e1358b7306bd3406e9330263a4. state=OPEN, ts=1365336963208, server=asf001.sp2.ygridcore.net,49518,1365336962323} to {testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,g,1365336962398.682475e1358b7306bd3406e9330263a4. state=OPEN, ts=1365336963257, server=asf001.sp2.ygridcore.net,49518,1365336962323} 2013-04-07 12:16:03,257 INFO [hbase-am-zkevent-worker-pool-2-thread-24] master.RegionStates(299): Onlined region {NAME => 'testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,g,1365336962398.682475e1358b7306bd3406e9330263a4.', STARTKEY => 'g', ENDKEY => 'h', ENCODED => 682475e1358b7306bd3406e9330263a4,} on asf001.sp2.ygridcore.net,49518,1365336962323 2013-04-07 12:16:03,257 INFO [hbase-am-zkevent-worker-pool-2-thread-24] master.AssignmentManager$4(1164): The master has opened the region testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,g,1365336962398.682475e1358b7306bd3406e9330263a4. that was online on asf001.sp2.ygridcore.net,49518,1365336962323 2013-04-07 12:16:03,258 DEBUG [hbase-am-zkevent-worker-pool-20-thread-22] master.AssignmentManager$4(1159): The znode of region testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,g,1365336962398.682475e1358b7306bd3406e9330263a4. has been deleted. 2013-04-07 12:16:03,258 INFO [hbase-am-zkevent-worker-pool-20-thread-22] master.RegionStates(264): Region {NAME => 'testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,g,1365336962398.682475e1358b7306bd3406e9330263a4.', STARTKEY => 'g', ENDKEY => 'h', ENCODED => 682475e1358b7306bd3406e9330263a4,} transitioned from {testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,g,1365336962398.682475e1358b7306bd3406e9330263a4. state=OPEN, ts=1365336963201, server=asf001.sp2.ygridcore.net,49518,1365336962323} to {testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,g,1365336962398.682475e1358b7306bd3406e9330263a4. state=OPEN, ts=1365336963258, server=asf001.sp2.ygridcore.net,49518,1365336962323} 2013-04-07 12:16:03,258 INFO [hbase-am-zkevent-worker-pool-20-thread-22] master.RegionStates(299): Onlined region {NAME => 'testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,g,1365336962398.682475e1358b7306bd3406e9330263a4.', STARTKEY => 'g', ENDKEY => 'h', ENCODED => 682475e1358b7306bd3406e9330263a4,} on asf001.sp2.ygridcore.net,49518,1365336962323 2013-04-07 12:16:03,258 INFO [hbase-am-zkevent-worker-pool-20-thread-22] master.AssignmentManager$4(1164): The master has opened the region testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,g,1365336962398.682475e1358b7306bd3406e9330263a4. that was online on asf001.sp2.ygridcore.net,49518,1365336962323 2013-04-07 12:16:03,258 DEBUG [hbase-am-zkevent-worker-pool-20-thread-22] master.AssignmentManager$4(1159): The znode of region testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,d,1365336962398.e99798d2c4d566ddb063efe03842457f. has been deleted. 2013-04-07 12:16:03,258 INFO [hbase-am-zkevent-worker-pool-20-thread-22] master.RegionStates(264): Region {NAME => 'testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,d,1365336962398.e99798d2c4d566ddb063efe03842457f.', STARTKEY => 'd', ENDKEY => 'e', ENCODED => e99798d2c4d566ddb063efe03842457f,} transitioned from {testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,d,1365336962398.e99798d2c4d566ddb063efe03842457f. state=OPEN, ts=1365336963202, server=asf001.sp2.ygridcore.net,49518,1365336962323} to {testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,d,1365336962398.e99798d2c4d566ddb063efe03842457f. state=OPEN, ts=1365336963258, server=asf001.sp2.ygridcore.net,49518,1365336962323} 2013-04-07 12:16:03,258 INFO [hbase-am-zkevent-worker-pool-20-thread-22] master.RegionStates(299): Onlined region {NAME => 'testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,d,1365336962398.e99798d2c4d566ddb063efe03842457f.', STARTKEY => 'd', ENDKEY => 'e', ENCODED => e99798d2c4d566ddb063efe03842457f,} on asf001.sp2.ygridcore.net,49518,1365336962323 2013-04-07 12:16:03,258 INFO [hbase-am-zkevent-worker-pool-20-thread-22] master.AssignmentManager$4(1164): The master has opened the region testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,d,1365336962398.e99798d2c4d566ddb063efe03842457f. that was online on asf001.sp2.ygridcore.net,49518,1365336962323 2013-04-07 12:16:03,259 DEBUG [MASTER_OPEN_REGION-asf001.sp2.ygridcore.net,33796,1365336878036-0] zookeeper.ZKUtil(1680): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Retrieved 156 byte(s) of data from znode /hbase/region-in-transition/1ce6d62427641e2a0e135d3d27b54b5d; data=region=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,h,1365336962398.1ce6d62427641e2a0e135d3d27b54b5d., state=RS_ZK_REGION_OPENED, servername=asf001.sp2.ygridcore.net,43802,1365336889418, createTime=1365336963163, payload.length=0 2013-04-07 12:16:03,261 WARN [MASTER_OPEN_REGION-asf001.sp2.ygridcore.net,33796,1365336878036-3] zookeeper.RecoverableZooKeeper(161): Node /hbase/region-in-transition/682475e1358b7306bd3406e9330263a4 already deleted, retry=false 2013-04-07 12:16:03,261 WARN [MASTER_OPEN_REGION-asf001.sp2.ygridcore.net,33796,1365336878036-3] handler.OpenedRegionHandler(132): The znode of the region testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,g,1365336962398.682475e1358b7306bd3406e9330263a4. would have already been deleted 2013-04-07 12:16:03,261 ERROR [MASTER_OPEN_REGION-asf001.sp2.ygridcore.net,33796,1365336878036-3] handler.OpenedRegionHandler(104): The znode of region testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,g,1365336962398.682475e1358b7306bd3406e9330263a4. could not be deleted. 2013-04-07 12:16:03,261 DEBUG [MASTER_OPEN_REGION-asf001.sp2.ygridcore.net,33796,1365336878036-3] handler.OpenedRegionHandler(146): Handling OPENED event for testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,j,1365336962398.38f0eb146fb473b19541deaada3419b9. from asf001.sp2.ygridcore.net,49518,1365336962323; deleting unassigned node 2013-04-07 12:16:03,261 DEBUG [MASTER_OPEN_REGION-asf001.sp2.ygridcore.net,33796,1365336878036-3] zookeeper.ZKAssign(405): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Deleting existing unassigned node for 38f0eb146fb473b19541deaada3419b9 that is in expected state RS_ZK_REGION_OPENED 2013-04-07 12:16:03,263 DEBUG [MASTER_OPEN_REGION-asf001.sp2.ygridcore.net,33796,1365336878036-4] zookeeper.ZKUtil(1680): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Retrieved 156 byte(s) of data from znode /hbase/region-in-transition/385e5846630b0cb4493758961a2c2457; data=region=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,i,1365336962398.385e5846630b0cb4493758961a2c2457., state=RS_ZK_REGION_OPENED, servername=asf001.sp2.ygridcore.net,60884,1365336878389, createTime=1365336963171, payload.length=0 2013-04-07 12:16:03,264 DEBUG [MASTER_OPEN_REGION-asf001.sp2.ygridcore.net,33796,1365336878036-1] zookeeper.ZKUtil(1680): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Retrieved 156 byte(s) of data from znode /hbase/region-in-transition/1ce6d62427641e2a0e135d3d27b54b5d; data=region=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,h,1365336962398.1ce6d62427641e2a0e135d3d27b54b5d., state=RS_ZK_REGION_OPENED, servername=asf001.sp2.ygridcore.net,43802,1365336889418, createTime=1365336963163, payload.length=0 2013-04-07 12:16:03,264 DEBUG [Thread-200-EventThread] zookeeper.ZooKeeperWatcher(274): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/region-in-transition/1ce6d62427641e2a0e135d3d27b54b5d 2013-04-07 12:16:03,265 DEBUG [hbase-am-zkevent-worker-pool-2-thread-36] master.AssignmentManager$4(1159): The znode of region testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,h,1365336962398.1ce6d62427641e2a0e135d3d27b54b5d. has been deleted. 2013-04-07 12:16:03,265 DEBUG [hbase-am-zkevent-worker-pool-20-thread-31] master.AssignmentManager$4(1159): The znode of region testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,h,1365336962398.1ce6d62427641e2a0e135d3d27b54b5d. has been deleted. 2013-04-07 12:16:03,265 INFO [hbase-am-zkevent-worker-pool-2-thread-36] master.RegionStates(264): Region {NAME => 'testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,h,1365336962398.1ce6d62427641e2a0e135d3d27b54b5d.', STARTKEY => 'h', ENDKEY => 'i', ENCODED => 1ce6d62427641e2a0e135d3d27b54b5d,} transitioned from {testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,h,1365336962398.1ce6d62427641e2a0e135d3d27b54b5d. state=OPEN, ts=1365336963209, server=asf001.sp2.ygridcore.net,43802,1365336889418} to {testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,h,1365336962398.1ce6d62427641e2a0e135d3d27b54b5d. state=OPEN, ts=1365336963265, server=asf001.sp2.ygridcore.net,43802,1365336889418} 2013-04-07 12:16:03,265 INFO [hbase-am-zkevent-worker-pool-20-thread-31] master.RegionStates(264): Region {NAME => 'testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,h,1365336962398.1ce6d62427641e2a0e135d3d27b54b5d.', STARTKEY => 'h', ENDKEY => 'i', ENCODED => 1ce6d62427641e2a0e135d3d27b54b5d,} transitioned from {testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,h,1365336962398.1ce6d62427641e2a0e135d3d27b54b5d. state=OPEN, ts=1365336963210, server=asf001.sp2.ygridcore.net,43802,1365336889418} to {testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,h,1365336962398.1ce6d62427641e2a0e135d3d27b54b5d. state=OPEN, ts=1365336963265, server=asf001.sp2.ygridcore.net,43802,1365336889418} 2013-04-07 12:16:03,265 INFO [hbase-am-zkevent-worker-pool-2-thread-36] master.RegionStates(299): Onlined region {NAME => 'testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,h,1365336962398.1ce6d62427641e2a0e135d3d27b54b5d.', STARTKEY => 'h', ENDKEY => 'i', ENCODED => 1ce6d62427641e2a0e135d3d27b54b5d,} on asf001.sp2.ygridcore.net,43802,1365336889418 2013-04-07 12:16:03,265 INFO [hbase-am-zkevent-worker-pool-20-thread-31] master.RegionStates(299): Onlined region {NAME => 'testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,h,1365336962398.1ce6d62427641e2a0e135d3d27b54b5d.', STARTKEY => 'h', ENDKEY => 'i', ENCODED => 1ce6d62427641e2a0e135d3d27b54b5d,} on asf001.sp2.ygridcore.net,43802,1365336889418 2013-04-07 12:16:03,265 INFO [hbase-am-zkevent-worker-pool-2-thread-36] master.AssignmentManager$4(1164): The master has opened the region testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,h,1365336962398.1ce6d62427641e2a0e135d3d27b54b5d. that was online on asf001.sp2.ygridcore.net,43802,1365336889418 2013-04-07 12:16:03,266 INFO [hbase-am-zkevent-worker-pool-20-thread-31] master.AssignmentManager$4(1164): The master has opened the region testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,h,1365336962398.1ce6d62427641e2a0e135d3d27b54b5d. that was online on asf001.sp2.ygridcore.net,43802,1365336889418 2013-04-07 12:16:03,265 DEBUG [MASTER_OPEN_REGION-asf001.sp2.ygridcore.net,33796,1365336878036-0] zookeeper.ZKAssign(434): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Successfully deleted unassigned node for region 1ce6d62427641e2a0e135d3d27b54b5d in expected state RS_ZK_REGION_OPENED 2013-04-07 12:16:03,265 DEBUG [Thread-200-EventThread] zookeeper.ZooKeeperWatcher(274): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/region-in-transition 2013-04-07 12:16:03,266 DEBUG [MASTER_OPEN_REGION-asf001.sp2.ygridcore.net,33796,1365336878036-0] handler.OpenedRegionHandler(146): Handling OPENED event for testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,j,1365336962398.38f0eb146fb473b19541deaada3419b9. from asf001.sp2.ygridcore.net,49518,1365336962323; deleting unassigned node 2013-04-07 12:16:03,267 DEBUG [MASTER_OPEN_REGION-asf001.sp2.ygridcore.net,33796,1365336878036-0] zookeeper.ZKAssign(405): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Deleting existing unassigned node for 38f0eb146fb473b19541deaada3419b9 that is in expected state RS_ZK_REGION_OPENED 2013-04-07 12:16:03,268 DEBUG [MASTER_OPEN_REGION-asf001.sp2.ygridcore.net,33796,1365336878036-2] zookeeper.ZKUtil(1680): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Retrieved 156 byte(s) of data from znode /hbase/region-in-transition/385e5846630b0cb4493758961a2c2457; data=region=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,i,1365336962398.385e5846630b0cb4493758961a2c2457., state=RS_ZK_REGION_OPENED, servername=asf001.sp2.ygridcore.net,60884,1365336878389, createTime=1365336963171, payload.length=0 2013-04-07 12:16:03,270 DEBUG [MASTER_OPEN_REGION-asf001.sp2.ygridcore.net,33796,1365336878036-3] zookeeper.ZKUtil(1680): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Retrieved 156 byte(s) of data from znode /hbase/region-in-transition/38f0eb146fb473b19541deaada3419b9; data=region=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,j,1365336962398.38f0eb146fb473b19541deaada3419b9., state=RS_ZK_REGION_OPENED, servername=asf001.sp2.ygridcore.net,49518,1365336962323, createTime=1365336963218, payload.length=0 2013-04-07 12:16:03,271 DEBUG [Thread-200-EventThread] zookeeper.ZooKeeperWatcher(274): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/region-in-transition/385e5846630b0cb4493758961a2c2457 2013-04-07 12:16:03,271 DEBUG [hbase-am-zkevent-worker-pool-2-thread-37] master.AssignmentManager$4(1159): The znode of region testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,i,1365336962398.385e5846630b0cb4493758961a2c2457. has been deleted. 2013-04-07 12:16:03,271 DEBUG [hbase-am-zkevent-worker-pool-20-thread-25] master.AssignmentManager$4(1159): The znode of region testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,i,1365336962398.385e5846630b0cb4493758961a2c2457. has been deleted. 2013-04-07 12:16:03,271 DEBUG [MASTER_OPEN_REGION-asf001.sp2.ygridcore.net,33796,1365336878036-4] zookeeper.ZKAssign(434): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Successfully deleted unassigned node for region 385e5846630b0cb4493758961a2c2457 in expected state RS_ZK_REGION_OPENED 2013-04-07 12:16:03,272 INFO [hbase-am-zkevent-worker-pool-20-thread-25] master.RegionStates(264): Region {NAME => 'testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,i,1365336962398.385e5846630b0cb4493758961a2c2457.', STARTKEY => 'i', ENDKEY => 'j', ENCODED => 385e5846630b0cb4493758961a2c2457,} transitioned from {testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,i,1365336962398.385e5846630b0cb4493758961a2c2457. state=OPEN, ts=1365336963220, server=asf001.sp2.ygridcore.net,60884,1365336878389} to {testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,i,1365336962398.385e5846630b0cb4493758961a2c2457. state=OPEN, ts=1365336963271, server=asf001.sp2.ygridcore.net,60884,1365336878389} 2013-04-07 12:16:03,272 INFO [hbase-am-zkevent-worker-pool-20-thread-25] master.RegionStates(299): Onlined region {NAME => 'testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,i,1365336962398.385e5846630b0cb4493758961a2c2457.', STARTKEY => 'i', ENDKEY => 'j', ENCODED => 385e5846630b0cb4493758961a2c2457,} on asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:16:03,272 INFO [hbase-am-zkevent-worker-pool-20-thread-25] master.AssignmentManager$4(1164): The master has opened the region testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,i,1365336962398.385e5846630b0cb4493758961a2c2457. that was online on asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:16:03,271 INFO [hbase-am-zkevent-worker-pool-2-thread-37] master.RegionStates(264): Region {NAME => 'testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,i,1365336962398.385e5846630b0cb4493758961a2c2457.', STARTKEY => 'i', ENDKEY => 'j', ENCODED => 385e5846630b0cb4493758961a2c2457,} transitioned from {testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,i,1365336962398.385e5846630b0cb4493758961a2c2457. state=OPEN, ts=1365336963223, server=asf001.sp2.ygridcore.net,60884,1365336878389} to {testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,i,1365336962398.385e5846630b0cb4493758961a2c2457. state=OPEN, ts=1365336963271, server=asf001.sp2.ygridcore.net,60884,1365336878389} 2013-04-07 12:16:03,272 WARN [MASTER_OPEN_REGION-asf001.sp2.ygridcore.net,33796,1365336878036-1] zookeeper.RecoverableZooKeeper(161): Node /hbase/region-in-transition/1ce6d62427641e2a0e135d3d27b54b5d already deleted, retry=false 2013-04-07 12:16:03,272 DEBUG [MASTER_OPEN_REGION-asf001.sp2.ygridcore.net,33796,1365336878036-4] handler.OpenedRegionHandler(146): Handling OPENED event for testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,b,1365336962398.11c8123e8d563f264d947ffa106e4c7c. from asf001.sp2.ygridcore.net,60884,1365336878389; deleting unassigned node 2013-04-07 12:16:03,272 WARN [MASTER_OPEN_REGION-asf001.sp2.ygridcore.net,33796,1365336878036-1] handler.OpenedRegionHandler(132): The znode of the region testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,h,1365336962398.1ce6d62427641e2a0e135d3d27b54b5d. would have already been deleted 2013-04-07 12:16:03,272 DEBUG [MASTER_OPEN_REGION-asf001.sp2.ygridcore.net,33796,1365336878036-4] zookeeper.ZKAssign(405): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Deleting existing unassigned node for 11c8123e8d563f264d947ffa106e4c7c that is in expected state RS_ZK_REGION_OPENED 2013-04-07 12:16:03,272 ERROR [MASTER_OPEN_REGION-asf001.sp2.ygridcore.net,33796,1365336878036-1] handler.OpenedRegionHandler(104): The znode of region testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,h,1365336962398.1ce6d62427641e2a0e135d3d27b54b5d. could not be deleted. 2013-04-07 12:16:03,272 INFO [hbase-am-zkevent-worker-pool-2-thread-37] master.RegionStates(299): Onlined region {NAME => 'testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,i,1365336962398.385e5846630b0cb4493758961a2c2457.', STARTKEY => 'i', ENDKEY => 'j', ENCODED => 385e5846630b0cb4493758961a2c2457,} on asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:16:03,273 INFO [hbase-am-zkevent-worker-pool-2-thread-37] master.AssignmentManager$4(1164): The master has opened the region testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,i,1365336962398.385e5846630b0cb4493758961a2c2457. that was online on asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:16:03,273 DEBUG [MASTER_OPEN_REGION-asf001.sp2.ygridcore.net,33796,1365336878036-1] handler.OpenedRegionHandler(146): Handling OPENED event for testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,b,1365336962398.11c8123e8d563f264d947ffa106e4c7c. from asf001.sp2.ygridcore.net,60884,1365336878389; deleting unassigned node 2013-04-07 12:16:03,273 DEBUG [MASTER_OPEN_REGION-asf001.sp2.ygridcore.net,33796,1365336878036-1] zookeeper.ZKAssign(405): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Deleting existing unassigned node for 11c8123e8d563f264d947ffa106e4c7c that is in expected state RS_ZK_REGION_OPENED 2013-04-07 12:16:03,275 DEBUG [MASTER_OPEN_REGION-asf001.sp2.ygridcore.net,33796,1365336878036-0] zookeeper.ZKUtil(1680): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Retrieved 156 byte(s) of data from znode /hbase/region-in-transition/38f0eb146fb473b19541deaada3419b9; data=region=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,j,1365336962398.38f0eb146fb473b19541deaada3419b9., state=RS_ZK_REGION_OPENED, servername=asf001.sp2.ygridcore.net,49518,1365336962323, createTime=1365336963218, payload.length=0 2013-04-07 12:16:03,275 WARN [MASTER_OPEN_REGION-asf001.sp2.ygridcore.net,33796,1365336878036-2] zookeeper.RecoverableZooKeeper(161): Node /hbase/region-in-transition/385e5846630b0cb4493758961a2c2457 already deleted, retry=false 2013-04-07 12:16:03,275 WARN [MASTER_OPEN_REGION-asf001.sp2.ygridcore.net,33796,1365336878036-2] handler.OpenedRegionHandler(132): The znode of the region testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,i,1365336962398.385e5846630b0cb4493758961a2c2457. would have already been deleted 2013-04-07 12:16:03,276 ERROR [MASTER_OPEN_REGION-asf001.sp2.ygridcore.net,33796,1365336878036-2] handler.OpenedRegionHandler(104): The znode of region testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,i,1365336962398.385e5846630b0cb4493758961a2c2457. could not be deleted. 2013-04-07 12:16:03,276 DEBUG [Thread-200-EventThread] zookeeper.ZooKeeperWatcher(274): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/region-in-transition/38f0eb146fb473b19541deaada3419b9 2013-04-07 12:16:03,276 DEBUG [Thread-200-EventThread] zookeeper.ZooKeeperWatcher(274): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/region-in-transition 2013-04-07 12:16:03,276 DEBUG [hbase-am-zkevent-worker-pool-20-thread-27] master.AssignmentManager$4(1159): The znode of region testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,j,1365336962398.38f0eb146fb473b19541deaada3419b9. has been deleted. 2013-04-07 12:16:03,276 INFO [hbase-am-zkevent-worker-pool-20-thread-27] master.RegionStates(264): Region {NAME => 'testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,j,1365336962398.38f0eb146fb473b19541deaada3419b9.', STARTKEY => 'j', ENDKEY => '', ENCODED => 38f0eb146fb473b19541deaada3419b9,} transitioned from {testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,j,1365336962398.38f0eb146fb473b19541deaada3419b9. state=OPEN, ts=1365336963242, server=asf001.sp2.ygridcore.net,49518,1365336962323} to {testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,j,1365336962398.38f0eb146fb473b19541deaada3419b9. state=OPEN, ts=1365336963276, server=asf001.sp2.ygridcore.net,49518,1365336962323} 2013-04-07 12:16:03,276 DEBUG [MASTER_OPEN_REGION-asf001.sp2.ygridcore.net,33796,1365336878036-3] zookeeper.ZKAssign(434): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Successfully deleted unassigned node for region 38f0eb146fb473b19541deaada3419b9 in expected state RS_ZK_REGION_OPENED 2013-04-07 12:16:03,276 INFO [hbase-am-zkevent-worker-pool-20-thread-27] master.RegionStates(299): Onlined region {NAME => 'testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,j,1365336962398.38f0eb146fb473b19541deaada3419b9.', STARTKEY => 'j', ENDKEY => '', ENCODED => 38f0eb146fb473b19541deaada3419b9,} on asf001.sp2.ygridcore.net,49518,1365336962323 2013-04-07 12:16:03,277 INFO [hbase-am-zkevent-worker-pool-20-thread-27] master.AssignmentManager$4(1164): The master has opened the region testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,j,1365336962398.38f0eb146fb473b19541deaada3419b9. that was online on asf001.sp2.ygridcore.net,49518,1365336962323 2013-04-07 12:16:03,277 DEBUG [MASTER_OPEN_REGION-asf001.sp2.ygridcore.net,33796,1365336878036-4] zookeeper.ZKUtil(1680): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Retrieved 156 byte(s) of data from znode /hbase/region-in-transition/11c8123e8d563f264d947ffa106e4c7c; data=region=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,b,1365336962398.11c8123e8d563f264d947ffa106e4c7c., state=RS_ZK_REGION_OPENED, servername=asf001.sp2.ygridcore.net,60884,1365336878389, createTime=1365336963214, payload.length=0 2013-04-07 12:16:03,278 DEBUG [MASTER_OPEN_REGION-asf001.sp2.ygridcore.net,33796,1365336878036-1] zookeeper.ZKUtil(1680): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Retrieved 156 byte(s) of data from znode /hbase/region-in-transition/11c8123e8d563f264d947ffa106e4c7c; data=region=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,b,1365336962398.11c8123e8d563f264d947ffa106e4c7c., state=RS_ZK_REGION_OPENED, servername=asf001.sp2.ygridcore.net,60884,1365336878389, createTime=1365336963214, payload.length=0 2013-04-07 12:16:03,279 WARN [MASTER_OPEN_REGION-asf001.sp2.ygridcore.net,33796,1365336878036-0] zookeeper.RecoverableZooKeeper(161): Node /hbase/region-in-transition/38f0eb146fb473b19541deaada3419b9 already deleted, retry=false 2013-04-07 12:16:03,279 WARN [MASTER_OPEN_REGION-asf001.sp2.ygridcore.net,33796,1365336878036-0] handler.OpenedRegionHandler(132): The znode of the region testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,j,1365336962398.38f0eb146fb473b19541deaada3419b9. would have already been deleted 2013-04-07 12:16:03,279 ERROR [MASTER_OPEN_REGION-asf001.sp2.ygridcore.net,33796,1365336878036-0] handler.OpenedRegionHandler(104): The znode of region testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,j,1365336962398.38f0eb146fb473b19541deaada3419b9. could not be deleted. 2013-04-07 12:16:03,314 DEBUG [Thread-200-EventThread] zookeeper.ZooKeeperWatcher(274): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/region-in-transition/11c8123e8d563f264d947ffa106e4c7c 2013-04-07 12:16:03,314 DEBUG [Thread-200-EventThread] zookeeper.ZooKeeperWatcher(274): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/region-in-transition 2013-04-07 12:16:03,314 DEBUG [hbase-am-zkevent-worker-pool-2-thread-22] master.AssignmentManager$4(1159): The znode of region testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,b,1365336962398.11c8123e8d563f264d947ffa106e4c7c. has been deleted. 2013-04-07 12:16:03,314 INFO [hbase-am-zkevent-worker-pool-2-thread-22] master.RegionStates(264): Region {NAME => 'testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,b,1365336962398.11c8123e8d563f264d947ffa106e4c7c.', STARTKEY => 'b', ENDKEY => 'c', ENCODED => 11c8123e8d563f264d947ffa106e4c7c,} transitioned from {testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,b,1365336962398.11c8123e8d563f264d947ffa106e4c7c. state=OPEN, ts=1365336963249, server=asf001.sp2.ygridcore.net,60884,1365336878389} to {testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,b,1365336962398.11c8123e8d563f264d947ffa106e4c7c. state=OPEN, ts=1365336963314, server=asf001.sp2.ygridcore.net,60884,1365336878389} 2013-04-07 12:16:03,315 INFO [hbase-am-zkevent-worker-pool-2-thread-22] master.RegionStates(299): Onlined region {NAME => 'testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,b,1365336962398.11c8123e8d563f264d947ffa106e4c7c.', STARTKEY => 'b', ENDKEY => 'c', ENCODED => 11c8123e8d563f264d947ffa106e4c7c,} on asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:16:03,315 INFO [hbase-am-zkevent-worker-pool-2-thread-22] master.AssignmentManager$4(1164): The master has opened the region testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,b,1365336962398.11c8123e8d563f264d947ffa106e4c7c. that was online on asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:16:03,315 DEBUG [MASTER_OPEN_REGION-asf001.sp2.ygridcore.net,33796,1365336878036-4] zookeeper.ZKAssign(434): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Successfully deleted unassigned node for region 11c8123e8d563f264d947ffa106e4c7c in expected state RS_ZK_REGION_OPENED 2013-04-07 12:16:03,316 WARN [MASTER_OPEN_REGION-asf001.sp2.ygridcore.net,33796,1365336878036-1] zookeeper.RecoverableZooKeeper(161): Node /hbase/region-in-transition/11c8123e8d563f264d947ffa106e4c7c already deleted, retry=false 2013-04-07 12:16:03,316 WARN [MASTER_OPEN_REGION-asf001.sp2.ygridcore.net,33796,1365336878036-1] handler.OpenedRegionHandler(132): The znode of the region testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,b,1365336962398.11c8123e8d563f264d947ffa106e4c7c. would have already been deleted 2013-04-07 12:16:03,316 ERROR [MASTER_OPEN_REGION-asf001.sp2.ygridcore.net,33796,1365336878036-1] handler.OpenedRegionHandler(104): The znode of region testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,b,1365336962398.11c8123e8d563f264d947ffa106e4c7c. could not be deleted. 2013-04-07 12:16:03,316 DEBUG [hbase-am-zkevent-worker-pool-20-thread-26] master.AssignmentManager$4(1159): The znode of region testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,b,1365336962398.11c8123e8d563f264d947ffa106e4c7c. has been deleted. 2013-04-07 12:16:03,316 INFO [hbase-am-zkevent-worker-pool-20-thread-26] master.RegionStates(264): Region {NAME => 'testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,b,1365336962398.11c8123e8d563f264d947ffa106e4c7c.', STARTKEY => 'b', ENDKEY => 'c', ENCODED => 11c8123e8d563f264d947ffa106e4c7c,} transitioned from {testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,b,1365336962398.11c8123e8d563f264d947ffa106e4c7c. state=OPEN, ts=1365336963247, server=asf001.sp2.ygridcore.net,60884,1365336878389} to {testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,b,1365336962398.11c8123e8d563f264d947ffa106e4c7c. state=OPEN, ts=1365336963316, server=asf001.sp2.ygridcore.net,60884,1365336878389} 2013-04-07 12:16:03,316 INFO [hbase-am-zkevent-worker-pool-20-thread-26] master.RegionStates(299): Onlined region {NAME => 'testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,b,1365336962398.11c8123e8d563f264d947ffa106e4c7c.', STARTKEY => 'b', ENDKEY => 'c', ENCODED => 11c8123e8d563f264d947ffa106e4c7c,} on asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:16:03,316 INFO [hbase-am-zkevent-worker-pool-20-thread-26] master.AssignmentManager$4(1164): The master has opened the region testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,b,1365336962398.11c8123e8d563f264d947ffa106e4c7c. that was online on asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:16:03,322 DEBUG [hbase-am-zkevent-worker-pool-2-thread-26] master.AssignmentManager$4(1159): The znode of region testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,j,1365336962398.38f0eb146fb473b19541deaada3419b9. has been deleted. 2013-04-07 12:16:03,322 INFO [hbase-am-zkevent-worker-pool-2-thread-26] master.RegionStates(264): Region {NAME => 'testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,j,1365336962398.38f0eb146fb473b19541deaada3419b9.', STARTKEY => 'j', ENDKEY => '', ENCODED => 38f0eb146fb473b19541deaada3419b9,} transitioned from {testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,j,1365336962398.38f0eb146fb473b19541deaada3419b9. state=OPEN, ts=1365336963242, server=asf001.sp2.ygridcore.net,49518,1365336962323} to {testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,j,1365336962398.38f0eb146fb473b19541deaada3419b9. state=OPEN, ts=1365336963322, server=asf001.sp2.ygridcore.net,49518,1365336962323} 2013-04-07 12:16:03,322 INFO [hbase-am-zkevent-worker-pool-2-thread-26] master.RegionStates(299): Onlined region {NAME => 'testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,j,1365336962398.38f0eb146fb473b19541deaada3419b9.', STARTKEY => 'j', ENDKEY => '', ENCODED => 38f0eb146fb473b19541deaada3419b9,} on asf001.sp2.ygridcore.net,49518,1365336962323 2013-04-07 12:16:03,322 INFO [hbase-am-zkevent-worker-pool-2-thread-26] master.AssignmentManager$4(1164): The master has opened the region testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,j,1365336962398.38f0eb146fb473b19541deaada3419b9. that was online on asf001.sp2.ygridcore.net,49518,1365336962323 2013-04-07 12:14:49,394 INFO [pool-1-thread-1] hbase.ResourceChecker(147): before: TestZooKeeper#testLogSplittingAfterMasterRecoveryDueToZKExpiry Thread=213, OpenFileDescriptor=343, MaxFileDescriptor=60000, SystemLoadAverage=320, ProcessCount=175, AvailableMemoryMB=7064, ConnectionCount=4 2013-04-07 12:14:49,413 DEBUG [Thread-214] client.HConnectionManager(2673): HConnection retries=100 2013-04-07 12:14:49,419 DEBUG [RegionServer:2;asf001.sp2.ygridcore.net,43802,1365336889418] zookeeper.ZKConfig(84): Skipped reading ZK properties file 'zoo.cfg' since 'hbase.config.read.zookeeper.config' was not set to true 2013-04-07 12:14:49,420 DEBUG [RegionServer:2;asf001.sp2.ygridcore.net,43802,1365336889418] zookeeper.ZKUtil(119): regionserver:43802 opening connection to ZooKeeper with ensemble (localhost:51597) 2013-04-07 12:14:49,421 INFO [RegionServer:2;asf001.sp2.ygridcore.net,43802,1365336889418] zookeeper.RecoverableZooKeeper(119): The identifier of this process is regionserver:43802 2013-04-07 12:14:49,692 DEBUG [RegionServer:2;asf001.sp2.ygridcore.net,43802,1365336889418-EventThread] zookeeper.ZooKeeperWatcher(274): regionserver:43802 Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2013-04-07 12:14:49,692 DEBUG [RegionServer:2;asf001.sp2.ygridcore.net,43802,1365336889418] zookeeper.ZKUtil(422): regionserver:43802 Set watcher on existing znode /hbase/master 2013-04-07 12:14:49,693 DEBUG [RegionServer:2;asf001.sp2.ygridcore.net,43802,1365336889418-EventThread] zookeeper.ZooKeeperWatcher(351): regionserver:43802-0x13de46b5f8b000f connected 2013-04-07 12:14:49,694 DEBUG [RegionServer:2;asf001.sp2.ygridcore.net,43802,1365336889418] zookeeper.ZKUtil(1680): regionserver:43802-0x13de46b5f8b000f Retrieved 43 byte(s) of data from znode /hbase/master and set watcher; PBUF\x0A%\x0A\x18asf001.sp2.y... 2013-04-07 12:14:49,695 DEBUG [RegionServer:2;asf001.sp2.ygridcore.net,43802,1365336889418] zookeeper.ZKUtil(422): regionserver:43802-0x13de46b5f8b000f Set watcher on existing znode /hbase/running 2013-04-07 12:14:49,696 DEBUG [RegionServer:2;asf001.sp2.ygridcore.net,43802,1365336889418] zookeeper.ZKUtil(1680): regionserver:43802-0x13de46b5f8b000f Retrieved 34 byte(s) of data from znode /hbase/running and set watcher; PBUF\x0A\x1CSun Apr 07 12:14:... 2013-04-07 12:14:49,697 DEBUG [RegionServer:2;asf001.sp2.ygridcore.net,43802,1365336889418] zookeeper.ZKConfig(84): Skipped reading ZK properties file 'zoo.cfg' since 'hbase.config.read.zookeeper.config' was not set to true 2013-04-07 12:14:49,697 DEBUG [RegionServer:2;asf001.sp2.ygridcore.net,43802,1365336889418] zookeeper.ZKUtil(119): hconnection-0x1c9f8fb opening connection to ZooKeeper with ensemble (localhost:51597) 2013-04-07 12:14:49,698 INFO [RegionServer:2;asf001.sp2.ygridcore.net,43802,1365336889418] zookeeper.RecoverableZooKeeper(119): The identifier of this process is hconnection-0x1c9f8fb 2013-04-07 12:14:49,705 DEBUG [RegionServer:2;asf001.sp2.ygridcore.net,43802,1365336889418-EventThread] zookeeper.ZooKeeperWatcher(274): hconnection-0x1c9f8fb Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2013-04-07 12:14:49,706 DEBUG [RegionServer:2;asf001.sp2.ygridcore.net,43802,1365336889418-EventThread] zookeeper.ZooKeeperWatcher(351): hconnection-0x1c9f8fb-0x13de46b5f8b0010 connected 2013-04-07 12:14:49,707 DEBUG [RegionServer:2;asf001.sp2.ygridcore.net,43802,1365336889418] zookeeper.ZKUtil(1680): hconnection-0x1c9f8fb-0x13de46b5f8b0010 Retrieved 42 byte(s) of data from znode /hbase/hbaseid; data=PBUF\x0A$cd034c7a-08a3-4d5e-8... 2013-04-07 12:14:49,707 INFO [RegionServer:2;asf001.sp2.ygridcore.net,43802,1365336889418] client.HConnectionManager$HConnectionImplementation(699): ClusterId is cd034c7a-08a3-4d5e-8eea-f9d6aa6dd983 2013-04-07 12:14:49,707 DEBUG [RegionServer:2;asf001.sp2.ygridcore.net,43802,1365336889418] catalog.CatalogTracker(192): Starting catalog tracker org.apache.hadoop.hbase.catalog.CatalogTracker@d8431c 2013-04-07 12:14:49,709 DEBUG [RegionServer:2;asf001.sp2.ygridcore.net,43802,1365336889418] zookeeper.ZKUtil(422): regionserver:43802-0x13de46b5f8b000f Set watcher on existing znode /hbase/meta-region-server 2013-04-07 12:14:49,711 DEBUG [RegionServer:2;asf001.sp2.ygridcore.net,43802,1365336889418] zookeeper.ZKUtil(1680): regionserver:43802-0x13de46b5f8b000f Retrieved 43 byte(s) of data from znode /hbase/meta-region-server and set watcher; asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:14:49,712 DEBUG [RegionServer:2;asf001.sp2.ygridcore.net,43802,1365336889418] zookeeper.ZKUtil(1680): regionserver:43802-0x13de46b5f8b000f Retrieved 42 byte(s) of data from znode /hbase/hbaseid; data=PBUF\x0A$cd034c7a-08a3-4d5e-8... 2013-04-07 12:14:49,713 INFO [RegionServer:2;asf001.sp2.ygridcore.net,43802,1365336889418] regionserver.HRegionServer(634): ClusterId : cd034c7a-08a3-4d5e-8eea-f9d6aa6dd983 2013-04-07 12:14:49,715 INFO [RegionServer:2;asf001.sp2.ygridcore.net,43802,1365336889418] zookeeper.RecoverableZooKeeper(460): Node /hbase/online-snapshot/acquired already exists and this is not a retry 2013-04-07 12:14:49,718 INFO [RegionServer:2;asf001.sp2.ygridcore.net,43802,1365336889418] regionserver.MemStoreFlusher(117): globalMemStoreLimit=675.6m, globalMemStoreLimitLowMark=591.1m, maxHeap=1.6g 2013-04-07 12:14:49,719 INFO [RegionServer:2;asf001.sp2.ygridcore.net,43802,1365336889418] regionserver.HRegionServer$CompactionChecker(1237): Runs every 16mins, 40sec 2013-04-07 12:14:49,719 INFO [RegionServer:2;asf001.sp2.ygridcore.net,43802,1365336889418] regionserver.HRegionServer(1737): Attempting connect to Master server at asf001.sp2.ygridcore.net,33796,1365336878036 2013-04-07 12:14:49,720 INFO [RegionServer:2;asf001.sp2.ygridcore.net,43802,1365336889418] regionserver.HRegionServer(1744): Connected to master at asf001.sp2.ygridcore.net/67.195.138.31:33796 2013-04-07 12:14:49,720 INFO [RegionServer:2;asf001.sp2.ygridcore.net,43802,1365336889418] regionserver.HRegionServer(1790): Telling master at asf001.sp2.ygridcore.net,33796,1365336878036 that we are up with port=43802, startcode=1365336889418 2013-04-07 12:14:49,721 INFO [IPC Server handler 3 on 33796] master.ServerManager(352): Registering server=asf001.sp2.ygridcore.net,43802,1365336889418 2013-04-07 12:14:49,722 DEBUG [RegionServer:2;asf001.sp2.ygridcore.net,43802,1365336889418] regionserver.HRegionServer(1084): Config from master: hbase.rootdir=hdfs://localhost:60405/user/jenkins/hbase 2013-04-07 12:14:49,723 DEBUG [RegionServer:2;asf001.sp2.ygridcore.net,43802,1365336889418] regionserver.HRegionServer(1084): Config from master: fs.default.name=hdfs://localhost:60405 2013-04-07 12:14:49,723 INFO [RegionServer:2;asf001.sp2.ygridcore.net,43802,1365336889418] regionserver.HRegionServer(1077): Master passed us hostname to use. Was=asf001.sp2.ygridcore.net, Now=asf001.sp2.ygridcore.net 2013-04-07 12:14:49,727 WARN [RegionServer:2;asf001.sp2.ygridcore.net,43802,1365336889418] hbase.ZNodeClearer(58): No filename given to save the znode used, it won't be saved (Environment variable HBASE_ZNODE_FILE is not set). 2013-04-07 12:14:49,735 DEBUG [RegionServer:2;asf001.sp2.ygridcore.net,43802,1365336889418] fs.HFileSystem(199): Starting addLocationsOrderInterceptor with class class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2013-04-07 12:14:49,736 INFO [RegionServer:2;asf001.sp2.ygridcore.net,43802,1365336889418] fs.HFileSystem(246): Added intercepting call to namenode#getBlockLocations 2013-04-07 12:14:49,736 DEBUG [RegionServer:2;asf001.sp2.ygridcore.net,43802,1365336889418] regionserver.HRegionServer(1300): logdir=hdfs://localhost:60405/user/jenkins/hbase/.logs/asf001.sp2.ygridcore.net,43802,1365336889418 2013-04-07 12:14:49,739 INFO [RegionServer:2;asf001.sp2.ygridcore.net,43802,1365336889418] wal.FSHLog(354): HLog configuration: blocksize=64 MB, rollsize=60.8 MB, enabled=true, optionallogflushinternal=1000ms 2013-04-07 12:14:49,772 DEBUG [RegionServer:2;asf001.sp2.ygridcore.net,43802,1365336889418] wal.SequenceFileLogWriter(189): using new createWriter -- HADOOP-6840 2013-04-07 12:14:49,772 DEBUG [RegionServer:2;asf001.sp2.ygridcore.net,43802,1365336889418] wal.SequenceFileLogWriter(193): Path=hdfs://localhost:60405/user/jenkins/hbase/.logs/asf001.sp2.ygridcore.net,43802,1365336889418/asf001.sp2.ygridcore.net%2C43802%2C1365336889418.1365336889763, compression=false 2013-04-07 12:14:49,773 INFO [RegionServer:2;asf001.sp2.ygridcore.net,43802,1365336889418] wal.FSHLog(552): Rolled log; new path=/user/jenkins/hbase/.logs/asf001.sp2.ygridcore.net,43802,1365336889418/asf001.sp2.ygridcore.net%2C43802%2C1365336889418.1365336889763 2013-04-07 12:14:49,773 INFO [RegionServer:2;asf001.sp2.ygridcore.net,43802,1365336889418] wal.FSHLog(448): Using getNumCurrentReplicas--HDFS-826 2013-04-07 12:14:49,773 DEBUG [RegionServer:2;asf001.sp2.ygridcore.net,43802,1365336889418] executor.ExecutorService(99): Starting executor service name=RS_OPEN_REGION-asf001.sp2.ygridcore.net,43802,1365336889418, corePoolSize=3, maxPoolSize=3 2013-04-07 12:14:49,773 DEBUG [RegionServer:2;asf001.sp2.ygridcore.net,43802,1365336889418] executor.ExecutorService(99): Starting executor service name=RS_OPEN_META-asf001.sp2.ygridcore.net,43802,1365336889418, corePoolSize=1, maxPoolSize=1 2013-04-07 12:14:49,774 DEBUG [RegionServer:2;asf001.sp2.ygridcore.net,43802,1365336889418] executor.ExecutorService(99): Starting executor service name=RS_CLOSE_REGION-asf001.sp2.ygridcore.net,43802,1365336889418, corePoolSize=3, maxPoolSize=3 2013-04-07 12:14:49,774 DEBUG [RegionServer:2;asf001.sp2.ygridcore.net,43802,1365336889418] executor.ExecutorService(99): Starting executor service name=RS_CLOSE_META-asf001.sp2.ygridcore.net,43802,1365336889418, corePoolSize=1, maxPoolSize=1 2013-04-07 12:14:49,780 INFO [RegionServer:2;asf001.sp2.ygridcore.net,43802,1365336889418] regionserver.HRegionServer(1115): Serving as asf001.sp2.ygridcore.net,43802,1365336889418, RPC listening on asf001.sp2.ygridcore.net/67.195.138.31:43802, sessionid=0x13de46b5f8b000f 2013-04-07 12:14:49,780 INFO [SplitLogWorker-asf001.sp2.ygridcore.net,43802,1365336889418] regionserver.SplitLogWorker(135): SplitLogWorker asf001.sp2.ygridcore.net,43802,1365336889418 starting 2013-04-07 12:14:49,780 DEBUG [RegionServer:2;asf001.sp2.ygridcore.net,43802,1365336889418] procedure.ZKProcedureMemberRpcs(341): Starting procedure member 'asf001.sp2.ygridcore.net,43802,1365336889418' 2013-04-07 12:14:49,780 DEBUG [RegionServer:2;asf001.sp2.ygridcore.net,43802,1365336889418] procedure.ZKProcedureMemberRpcs(140): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2013-04-07 12:14:49,784 DEBUG [RegionServer:2;asf001.sp2.ygridcore.net,43802,1365336889418] procedure.ZKProcedureMemberRpcs(156): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2013-04-07 12:14:49,787 DEBUG [IPC Server handler 0 on 33796] zookeeper.ZKUtil(1680): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Retrieved 43 byte(s) of data from znode /hbase/meta-region-server and set watcher; asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:14:49,787 DEBUG [IPC Server handler 0 on 33796] master.TableLockManager$ZKTableLockManager$TableLockImpl(225): Attempt to acquire table write lock on :testLogSplittingAfterMasterRecoveryDueToZKExpiry for:C_M_CREATE_TABLE 2013-04-07 12:14:49,803 DEBUG [IPC Server handler 0 on 33796] zookeeper.ZKUtil(1680): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Retrieved 115 byte(s) of data from znode /hbase/table-lock/testLogSplittingAfterMasterRecoveryDueToZKExpiry/write-master:337960000000000; data=PBUF\x0A0testLogSplittingAfte... 2013-04-07 12:14:49,803 DEBUG [IPC Server handler 0 on 33796] lock.ZKInterProcessLockBase(220): Successfully acquired a lock for /hbase/table-lock/testLogSplittingAfterMasterRecoveryDueToZKExpiry/write-master:337960000000000 2013-04-07 12:14:49,803 DEBUG [IPC Server handler 0 on 33796] master.TableLockManager$ZKTableLockManager$TableLockImpl(246): Acquired table write lock on :testLogSplittingAfterMasterRecoveryDueToZKExpiry for:C_M_CREATE_TABLE 2013-04-07 12:14:49,805 DEBUG [IPC Server handler 0 on 33796] zookeeper.ZKUtil(1680): hconnection-0x1f327e-0x13de46b5f8b0003 Retrieved 43 byte(s) of data from znode /hbase/meta-region-server; data=asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:14:49,805 DEBUG [IPC Server handler 0 on 33796] client.ClientScanner(96): Creating scanner over .META. starting at key 'testLogSplittingAfterMasterRecoveryDueToZKExpiry,,' 2013-04-07 12:14:49,805 DEBUG [IPC Server handler 0 on 33796] client.ClientScanner(209): Advancing internal scanner to startKey at 'testLogSplittingAfterMasterRecoveryDueToZKExpiry,,' 2013-04-07 12:14:49,806 DEBUG [IPC Server handler 0 on 33796] zookeeper.ZKUtil(1680): hconnection-0x1f327e-0x13de46b5f8b0003 Retrieved 43 byte(s) of data from znode /hbase/meta-region-server; data=asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:14:49,811 DEBUG [IPC Server handler 0 on 33796] client.ClientScanner(196): Finished with scanning at {NAME => '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192,} 2013-04-07 12:14:49,817 INFO [MASTER_TABLE_OPERATIONS-asf001.sp2.ygridcore.net,33796,1365336878036-0] handler.CreateTableHandler(146): Attempting to create the table testLogSplittingAfterMasterRecoveryDueToZKExpiry 2013-04-07 12:14:49,818 DEBUG [Thread-214] client.MetaScanner(198): Scanning .META. starting at row=testLogSplittingAfterMasterRecoveryDueToZKExpiry,,00000000000000 for max=2147483647 rows using hconnection-0xfe04c6 2013-04-07 12:14:49,820 DEBUG [Thread-214] zookeeper.ZKUtil(1680): hconnection-0xfe04c6-0x13de46b5f8b0007 Retrieved 43 byte(s) of data from znode /hbase/meta-region-server; data=asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:14:50,264 INFO [RegionOpenAndInitThread-testLogSplittingAfterMasterRecoveryDueToZKExpiry-1] regionserver.HRegion(3862): creating HRegion testLogSplittingAfterMasterRecoveryDueToZKExpiry HTD == 'testLogSplittingAfterMasterRecoveryDueToZKExpiry', {NAME => 'col', DATA_BLOCK_ENCODING => 'NONE', BLOOMFILTER => 'NONE', REPLICATION_SCOPE => '0', VERSIONS => '3', COMPRESSION => 'NONE', MIN_VERSIONS => '0', TTL => '2147483647', KEEP_DELETED_CELLS => 'false', BLOCKSIZE => '65536', IN_MEMORY => 'false', ENCODE_ON_DISK => 'true', BLOCKCACHE => 'true'} RootDir = hdfs://localhost:60405/user/jenkins/hbase/.tmp Table name == testLogSplittingAfterMasterRecoveryDueToZKExpiry 2013-04-07 12:14:50,264 INFO [RegionOpenAndInitThread-testLogSplittingAfterMasterRecoveryDueToZKExpiry-2] regionserver.HRegion(3862): creating HRegion testLogSplittingAfterMasterRecoveryDueToZKExpiry HTD == 'testLogSplittingAfterMasterRecoveryDueToZKExpiry', {NAME => 'col', DATA_BLOCK_ENCODING => 'NONE', BLOOMFILTER => 'NONE', REPLICATION_SCOPE => '0', VERSIONS => '3', COMPRESSION => 'NONE', MIN_VERSIONS => '0', TTL => '2147483647', KEEP_DELETED_CELLS => 'false', BLOCKSIZE => '65536', IN_MEMORY => 'false', ENCODE_ON_DISK => 'true', BLOCKCACHE => 'true'} RootDir = hdfs://localhost:60405/user/jenkins/hbase/.tmp Table name == testLogSplittingAfterMasterRecoveryDueToZKExpiry 2013-04-07 12:14:50,266 INFO [RegionOpenAndInitThread-testLogSplittingAfterMasterRecoveryDueToZKExpiry-3] regionserver.HRegion(3862): creating HRegion testLogSplittingAfterMasterRecoveryDueToZKExpiry HTD == 'testLogSplittingAfterMasterRecoveryDueToZKExpiry', {NAME => 'col', DATA_BLOCK_ENCODING => 'NONE', BLOOMFILTER => 'NONE', REPLICATION_SCOPE => '0', VERSIONS => '3', COMPRESSION => 'NONE', MIN_VERSIONS => '0', TTL => '2147483647', KEEP_DELETED_CELLS => 'false', BLOCKSIZE => '65536', IN_MEMORY => 'false', ENCODE_ON_DISK => 'true', BLOCKCACHE => 'true'} RootDir = hdfs://localhost:60405/user/jenkins/hbase/.tmp Table name == testLogSplittingAfterMasterRecoveryDueToZKExpiry 2013-04-07 12:14:50,273 INFO [RegionOpenAndInitThread-testLogSplittingAfterMasterRecoveryDueToZKExpiry-5] regionserver.HRegion(3862): creating HRegion testLogSplittingAfterMasterRecoveryDueToZKExpiry HTD == 'testLogSplittingAfterMasterRecoveryDueToZKExpiry', {NAME => 'col', DATA_BLOCK_ENCODING => 'NONE', BLOOMFILTER => 'NONE', REPLICATION_SCOPE => '0', VERSIONS => '3', COMPRESSION => 'NONE', MIN_VERSIONS => '0', TTL => '2147483647', KEEP_DELETED_CELLS => 'false', BLOCKSIZE => '65536', IN_MEMORY => 'false', ENCODE_ON_DISK => 'true', BLOCKCACHE => 'true'} RootDir = hdfs://localhost:60405/user/jenkins/hbase/.tmp Table name == testLogSplittingAfterMasterRecoveryDueToZKExpiry 2013-04-07 12:14:50,273 INFO [RegionOpenAndInitThread-testLogSplittingAfterMasterRecoveryDueToZKExpiry-6] regionserver.HRegion(3862): creating HRegion testLogSplittingAfterMasterRecoveryDueToZKExpiry HTD == 'testLogSplittingAfterMasterRecoveryDueToZKExpiry', {NAME => 'col', DATA_BLOCK_ENCODING => 'NONE', BLOOMFILTER => 'NONE', REPLICATION_SCOPE => '0', VERSIONS => '3', COMPRESSION => 'NONE', MIN_VERSIONS => '0', TTL => '2147483647', KEEP_DELETED_CELLS => 'false', BLOCKSIZE => '65536', IN_MEMORY => 'false', ENCODE_ON_DISK => 'true', BLOCKCACHE => 'true'} RootDir = hdfs://localhost:60405/user/jenkins/hbase/.tmp Table name == testLogSplittingAfterMasterRecoveryDueToZKExpiry 2013-04-07 12:14:50,266 INFO [RegionOpenAndInitThread-testLogSplittingAfterMasterRecoveryDueToZKExpiry-4] regionserver.HRegion(3862): creating HRegion testLogSplittingAfterMasterRecoveryDueToZKExpiry HTD == 'testLogSplittingAfterMasterRecoveryDueToZKExpiry', {NAME => 'col', DATA_BLOCK_ENCODING => 'NONE', BLOOMFILTER => 'NONE', REPLICATION_SCOPE => '0', VERSIONS => '3', COMPRESSION => 'NONE', MIN_VERSIONS => '0', TTL => '2147483647', KEEP_DELETED_CELLS => 'false', BLOCKSIZE => '65536', IN_MEMORY => 'false', ENCODE_ON_DISK => 'true', BLOCKCACHE => 'true'} RootDir = hdfs://localhost:60405/user/jenkins/hbase/.tmp Table name == testLogSplittingAfterMasterRecoveryDueToZKExpiry 2013-04-07 12:14:50,277 DEBUG [RegionOpenAndInitThread-testLogSplittingAfterMasterRecoveryDueToZKExpiry-1] util.FSUtils(211): Creating file=hdfs://localhost:60405/user/jenkins/hbase/.tmp/testLogSplittingAfterMasterRecoveryDueToZKExpiry/fb4182aef4ce07f011871ae0a083aee0/.regioninfo with permission=rwxrwxrwx 2013-04-07 12:14:50,280 DEBUG [RegionOpenAndInitThread-testLogSplittingAfterMasterRecoveryDueToZKExpiry-2] util.FSUtils(211): Creating file=hdfs://localhost:60405/user/jenkins/hbase/.tmp/testLogSplittingAfterMasterRecoveryDueToZKExpiry/43207168ac66dce85cf2a4f916007450/.regioninfo with permission=rwxrwxrwx 2013-04-07 12:14:50,282 DEBUG [RegionOpenAndInitThread-testLogSplittingAfterMasterRecoveryDueToZKExpiry-5] util.FSUtils(211): Creating file=hdfs://localhost:60405/user/jenkins/hbase/.tmp/testLogSplittingAfterMasterRecoveryDueToZKExpiry/567d8ef20f90f3c1b8a889d22a1c3948/.regioninfo with permission=rwxrwxrwx 2013-04-07 12:14:50,282 DEBUG [RegionOpenAndInitThread-testLogSplittingAfterMasterRecoveryDueToZKExpiry-6] util.FSUtils(211): Creating file=hdfs://localhost:60405/user/jenkins/hbase/.tmp/testLogSplittingAfterMasterRecoveryDueToZKExpiry/5403a557a6168e5f3c5141eb494f6088/.regioninfo with permission=rwxrwxrwx 2013-04-07 12:14:50,283 DEBUG [RegionOpenAndInitThread-testLogSplittingAfterMasterRecoveryDueToZKExpiry-4] util.FSUtils(211): Creating file=hdfs://localhost:60405/user/jenkins/hbase/.tmp/testLogSplittingAfterMasterRecoveryDueToZKExpiry/4e9a3d63237d29e6bf55bf2360834cbe/.regioninfo with permission=rwxrwxrwx 2013-04-07 12:14:50,284 DEBUG [RegionOpenAndInitThread-testLogSplittingAfterMasterRecoveryDueToZKExpiry-3] util.FSUtils(211): Creating file=hdfs://localhost:60405/user/jenkins/hbase/.tmp/testLogSplittingAfterMasterRecoveryDueToZKExpiry/55884b67b9203a71bdf8c0e3de70452d/.regioninfo with permission=rwxrwxrwx 2013-04-07 12:14:50,317 DEBUG [RegionOpenAndInitThread-testLogSplittingAfterMasterRecoveryDueToZKExpiry-5] regionserver.HRegion(496): Instantiated testLogSplittingAfterMasterRecoveryDueToZKExpiry,4,1365336889784.567d8ef20f90f3c1b8a889d22a1c3948. 2013-04-07 12:14:50,325 DEBUG [RegionOpenAndInitThread-testLogSplittingAfterMasterRecoveryDueToZKExpiry-5] regionserver.HRegion(889): Closing testLogSplittingAfterMasterRecoveryDueToZKExpiry,4,1365336889784.567d8ef20f90f3c1b8a889d22a1c3948.: disabling compactions & flushes 2013-04-07 12:14:50,325 DEBUG [RegionOpenAndInitThread-testLogSplittingAfterMasterRecoveryDueToZKExpiry-5] regionserver.HRegion(911): Updates disabled for region testLogSplittingAfterMasterRecoveryDueToZKExpiry,4,1365336889784.567d8ef20f90f3c1b8a889d22a1c3948. 2013-04-07 12:14:50,325 INFO [RegionOpenAndInitThread-testLogSplittingAfterMasterRecoveryDueToZKExpiry-5] regionserver.HRegion(968): Closed testLogSplittingAfterMasterRecoveryDueToZKExpiry,4,1365336889784.567d8ef20f90f3c1b8a889d22a1c3948. 2013-04-07 12:14:50,325 DEBUG [RegionOpenAndInitThread-testLogSplittingAfterMasterRecoveryDueToZKExpiry-1] regionserver.HRegion(496): Instantiated testLogSplittingAfterMasterRecoveryDueToZKExpiry,,1365336889784.fb4182aef4ce07f011871ae0a083aee0. 2013-04-07 12:14:50,326 DEBUG [RegionOpenAndInitThread-testLogSplittingAfterMasterRecoveryDueToZKExpiry-1] regionserver.HRegion(889): Closing testLogSplittingAfterMasterRecoveryDueToZKExpiry,,1365336889784.fb4182aef4ce07f011871ae0a083aee0.: disabling compactions & flushes 2013-04-07 12:14:50,326 DEBUG [RegionOpenAndInitThread-testLogSplittingAfterMasterRecoveryDueToZKExpiry-1] regionserver.HRegion(911): Updates disabled for region testLogSplittingAfterMasterRecoveryDueToZKExpiry,,1365336889784.fb4182aef4ce07f011871ae0a083aee0. 2013-04-07 12:14:50,326 INFO [RegionOpenAndInitThread-testLogSplittingAfterMasterRecoveryDueToZKExpiry-1] regionserver.HRegion(968): Closed testLogSplittingAfterMasterRecoveryDueToZKExpiry,,1365336889784.fb4182aef4ce07f011871ae0a083aee0. 2013-04-07 12:14:50,328 DEBUG [RegionOpenAndInitThread-testLogSplittingAfterMasterRecoveryDueToZKExpiry-2] regionserver.HRegion(496): Instantiated testLogSplittingAfterMasterRecoveryDueToZKExpiry,1,1365336889784.43207168ac66dce85cf2a4f916007450. 2013-04-07 12:14:50,328 DEBUG [RegionOpenAndInitThread-testLogSplittingAfterMasterRecoveryDueToZKExpiry-2] regionserver.HRegion(889): Closing testLogSplittingAfterMasterRecoveryDueToZKExpiry,1,1365336889784.43207168ac66dce85cf2a4f916007450.: disabling compactions & flushes 2013-04-07 12:14:50,350 DEBUG [RegionOpenAndInitThread-testLogSplittingAfterMasterRecoveryDueToZKExpiry-2] regionserver.HRegion(911): Updates disabled for region testLogSplittingAfterMasterRecoveryDueToZKExpiry,1,1365336889784.43207168ac66dce85cf2a4f916007450. 2013-04-07 12:14:50,351 INFO [RegionOpenAndInitThread-testLogSplittingAfterMasterRecoveryDueToZKExpiry-2] regionserver.HRegion(968): Closed testLogSplittingAfterMasterRecoveryDueToZKExpiry,1,1365336889784.43207168ac66dce85cf2a4f916007450. 2013-04-07 12:14:50,353 DEBUG [RegionOpenAndInitThread-testLogSplittingAfterMasterRecoveryDueToZKExpiry-4] regionserver.HRegion(496): Instantiated testLogSplittingAfterMasterRecoveryDueToZKExpiry,3,1365336889784.4e9a3d63237d29e6bf55bf2360834cbe. 2013-04-07 12:14:50,353 DEBUG [RegionOpenAndInitThread-testLogSplittingAfterMasterRecoveryDueToZKExpiry-4] regionserver.HRegion(889): Closing testLogSplittingAfterMasterRecoveryDueToZKExpiry,3,1365336889784.4e9a3d63237d29e6bf55bf2360834cbe.: disabling compactions & flushes 2013-04-07 12:14:50,354 DEBUG [RegionOpenAndInitThread-testLogSplittingAfterMasterRecoveryDueToZKExpiry-4] regionserver.HRegion(911): Updates disabled for region testLogSplittingAfterMasterRecoveryDueToZKExpiry,3,1365336889784.4e9a3d63237d29e6bf55bf2360834cbe. 2013-04-07 12:14:50,354 INFO [RegionOpenAndInitThread-testLogSplittingAfterMasterRecoveryDueToZKExpiry-4] regionserver.HRegion(968): Closed testLogSplittingAfterMasterRecoveryDueToZKExpiry,3,1365336889784.4e9a3d63237d29e6bf55bf2360834cbe. 2013-04-07 12:14:50,358 DEBUG [RegionOpenAndInitThread-testLogSplittingAfterMasterRecoveryDueToZKExpiry-3] regionserver.HRegion(496): Instantiated testLogSplittingAfterMasterRecoveryDueToZKExpiry,2,1365336889784.55884b67b9203a71bdf8c0e3de70452d. 2013-04-07 12:14:50,358 DEBUG [RegionOpenAndInitThread-testLogSplittingAfterMasterRecoveryDueToZKExpiry-3] regionserver.HRegion(889): Closing testLogSplittingAfterMasterRecoveryDueToZKExpiry,2,1365336889784.55884b67b9203a71bdf8c0e3de70452d.: disabling compactions & flushes 2013-04-07 12:14:50,358 DEBUG [RegionOpenAndInitThread-testLogSplittingAfterMasterRecoveryDueToZKExpiry-3] regionserver.HRegion(911): Updates disabled for region testLogSplittingAfterMasterRecoveryDueToZKExpiry,2,1365336889784.55884b67b9203a71bdf8c0e3de70452d. 2013-04-07 12:14:50,358 INFO [RegionOpenAndInitThread-testLogSplittingAfterMasterRecoveryDueToZKExpiry-3] regionserver.HRegion(968): Closed testLogSplittingAfterMasterRecoveryDueToZKExpiry,2,1365336889784.55884b67b9203a71bdf8c0e3de70452d. 2013-04-07 12:14:50,399 DEBUG [RegionOpenAndInitThread-testLogSplittingAfterMasterRecoveryDueToZKExpiry-6] regionserver.HRegion(496): Instantiated testLogSplittingAfterMasterRecoveryDueToZKExpiry,5,1365336889784.5403a557a6168e5f3c5141eb494f6088. 2013-04-07 12:14:50,399 DEBUG [RegionOpenAndInitThread-testLogSplittingAfterMasterRecoveryDueToZKExpiry-6] regionserver.HRegion(889): Closing testLogSplittingAfterMasterRecoveryDueToZKExpiry,5,1365336889784.5403a557a6168e5f3c5141eb494f6088.: disabling compactions & flushes 2013-04-07 12:14:50,399 DEBUG [RegionOpenAndInitThread-testLogSplittingAfterMasterRecoveryDueToZKExpiry-6] regionserver.HRegion(911): Updates disabled for region testLogSplittingAfterMasterRecoveryDueToZKExpiry,5,1365336889784.5403a557a6168e5f3c5141eb494f6088. 2013-04-07 12:14:50,399 INFO [RegionOpenAndInitThread-testLogSplittingAfterMasterRecoveryDueToZKExpiry-6] regionserver.HRegion(968): Closed testLogSplittingAfterMasterRecoveryDueToZKExpiry,5,1365336889784.5403a557a6168e5f3c5141eb494f6088. 2013-04-07 12:14:50,410 DEBUG [MASTER_TABLE_OPERATIONS-asf001.sp2.ygridcore.net,33796,1365336878036-0] zookeeper.ZKUtil(1680): hconnection-0x1f327e-0x13de46b5f8b0003 Retrieved 43 byte(s) of data from znode /hbase/meta-region-server; data=asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:14:50,411 DEBUG [MASTER_TABLE_OPERATIONS-asf001.sp2.ygridcore.net,33796,1365336878036-0] zookeeper.ZKUtil(1680): hconnection-0x1f327e-0x13de46b5f8b0003 Retrieved 43 byte(s) of data from znode /hbase/meta-region-server; data=asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:14:50,412 DEBUG [MASTER_TABLE_OPERATIONS-asf001.sp2.ygridcore.net,33796,1365336878036-0] zookeeper.ZKUtil(1680): hconnection-0x1f327e-0x13de46b5f8b0003 Retrieved 43 byte(s) of data from znode /hbase/meta-region-server; data=asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:14:50,413 DEBUG [MASTER_TABLE_OPERATIONS-asf001.sp2.ygridcore.net,33796,1365336878036-0] zookeeper.ZKUtil(1680): hconnection-0x1f327e-0x13de46b5f8b0003 Retrieved 43 byte(s) of data from znode /hbase/meta-region-server; data=asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:14:50,414 DEBUG [MASTER_TABLE_OPERATIONS-asf001.sp2.ygridcore.net,33796,1365336878036-0] zookeeper.ZKUtil(1680): hconnection-0x1f327e-0x13de46b5f8b0003 Retrieved 43 byte(s) of data from znode /hbase/meta-region-server; data=asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:14:50,416 DEBUG [MASTER_TABLE_OPERATIONS-asf001.sp2.ygridcore.net,33796,1365336878036-0] zookeeper.ZKUtil(1680): hconnection-0x1f327e-0x13de46b5f8b0003 Retrieved 43 byte(s) of data from znode /hbase/meta-region-server; data=asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:14:50,417 DEBUG [MASTER_TABLE_OPERATIONS-asf001.sp2.ygridcore.net,33796,1365336878036-0] zookeeper.ZKUtil(1680): hconnection-0x1f327e-0x13de46b5f8b0003 Retrieved 43 byte(s) of data from znode /hbase/meta-region-server; data=asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:14:50,422 INFO [MASTER_TABLE_OPERATIONS-asf001.sp2.ygridcore.net,33796,1365336878036-0] catalog.MetaEditor(254): Added 6 regions in META 2013-04-07 12:14:50,423 INFO [MASTER_TABLE_OPERATIONS-asf001.sp2.ygridcore.net,33796,1365336878036-0] master.AssignmentManager(2363): Not use bulk assigning since we are assigning only 6 region(s) to 2 server(s) 2013-04-07 12:14:50,423 DEBUG [MASTER_TABLE_OPERATIONS-asf001.sp2.ygridcore.net,33796,1365336878036-0] master.AssignmentManager(1444): Bulk assigning 3 region(s) to asf001.sp2.ygridcore.net,43802,1365336889418 2013-04-07 12:14:50,423 INFO [MASTER_TABLE_OPERATIONS-asf001.sp2.ygridcore.net,33796,1365336878036-0] master.RegionStates(264): Region {NAME => 'testLogSplittingAfterMasterRecoveryDueToZKExpiry,4,1365336889784.567d8ef20f90f3c1b8a889d22a1c3948.', STARTKEY => '4', ENDKEY => '5', ENCODED => 567d8ef20f90f3c1b8a889d22a1c3948,} transitioned from {testLogSplittingAfterMasterRecoveryDueToZKExpiry,4,1365336889784.567d8ef20f90f3c1b8a889d22a1c3948. state=OFFLINE, ts=1365336890422, server=null} to {testLogSplittingAfterMasterRecoveryDueToZKExpiry,4,1365336889784.567d8ef20f90f3c1b8a889d22a1c3948. state=OFFLINE, ts=1365336890423, server=null} 2013-04-07 12:14:50,423 DEBUG [MASTER_TABLE_OPERATIONS-asf001.sp2.ygridcore.net,33796,1365336878036-0] zookeeper.ZKAssign(177): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Async create of unassigned node for 567d8ef20f90f3c1b8a889d22a1c3948 with OFFLINE state 2013-04-07 12:14:50,423 INFO [MASTER_TABLE_OPERATIONS-asf001.sp2.ygridcore.net,33796,1365336878036-0] master.RegionStates(264): Region {NAME => 'testLogSplittingAfterMasterRecoveryDueToZKExpiry,1,1365336889784.43207168ac66dce85cf2a4f916007450.', STARTKEY => '1', ENDKEY => '2', ENCODED => 43207168ac66dce85cf2a4f916007450,} transitioned from {testLogSplittingAfterMasterRecoveryDueToZKExpiry,1,1365336889784.43207168ac66dce85cf2a4f916007450. state=OFFLINE, ts=1365336890422, server=null} to {testLogSplittingAfterMasterRecoveryDueToZKExpiry,1,1365336889784.43207168ac66dce85cf2a4f916007450. state=OFFLINE, ts=1365336890423, server=null} 2013-04-07 12:14:50,423 DEBUG [MASTER_TABLE_OPERATIONS-asf001.sp2.ygridcore.net,33796,1365336878036-0] zookeeper.ZKAssign(177): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Async create of unassigned node for 43207168ac66dce85cf2a4f916007450 with OFFLINE state 2013-04-07 12:14:50,424 INFO [MASTER_TABLE_OPERATIONS-asf001.sp2.ygridcore.net,33796,1365336878036-0] master.RegionStates(264): Region {NAME => 'testLogSplittingAfterMasterRecoveryDueToZKExpiry,2,1365336889784.55884b67b9203a71bdf8c0e3de70452d.', STARTKEY => '2', ENDKEY => '3', ENCODED => 55884b67b9203a71bdf8c0e3de70452d,} transitioned from {testLogSplittingAfterMasterRecoveryDueToZKExpiry,2,1365336889784.55884b67b9203a71bdf8c0e3de70452d. state=OFFLINE, ts=1365336890423, server=null} to {testLogSplittingAfterMasterRecoveryDueToZKExpiry,2,1365336889784.55884b67b9203a71bdf8c0e3de70452d. state=OFFLINE, ts=1365336890424, server=null} 2013-04-07 12:14:50,424 DEBUG [MASTER_TABLE_OPERATIONS-asf001.sp2.ygridcore.net,33796,1365336878036-0] zookeeper.ZKAssign(177): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Async create of unassigned node for 55884b67b9203a71bdf8c0e3de70452d with OFFLINE state 2013-04-07 12:14:50,462 INFO [PRI IPC Server handler 0 on 43802] regionserver.HRegionServer(3346): Received request to open region: testLogSplittingAfterMasterRecoveryDueToZKExpiry,4,1365336889784.567d8ef20f90f3c1b8a889d22a1c3948. on asf001.sp2.ygridcore.net,43802,1365336889418 2013-04-07 12:14:50,468 INFO [PRI IPC Server handler 0 on 43802] regionserver.HRegionServer(3346): Received request to open region: testLogSplittingAfterMasterRecoveryDueToZKExpiry,1,1365336889784.43207168ac66dce85cf2a4f916007450. on asf001.sp2.ygridcore.net,43802,1365336889418 2013-04-07 12:14:50,469 DEBUG [RS_OPEN_REGION-asf001.sp2.ygridcore.net,43802,1365336889418-0] zookeeper.ZKAssign(721): regionserver:43802-0x13de46b5f8b000f Attempting to transition node 567d8ef20f90f3c1b8a889d22a1c3948 from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING 2013-04-07 12:14:50,469 INFO [PRI IPC Server handler 0 on 43802] regionserver.HRegionServer(3346): Received request to open region: testLogSplittingAfterMasterRecoveryDueToZKExpiry,2,1365336889784.55884b67b9203a71bdf8c0e3de70452d. on asf001.sp2.ygridcore.net,43802,1365336889418 2013-04-07 12:14:50,469 DEBUG [RS_OPEN_REGION-asf001.sp2.ygridcore.net,43802,1365336889418-1] zookeeper.ZKAssign(721): regionserver:43802-0x13de46b5f8b000f Attempting to transition node 43207168ac66dce85cf2a4f916007450 from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING 2013-04-07 12:14:50,470 DEBUG [RS_OPEN_REGION-asf001.sp2.ygridcore.net,43802,1365336889418-2] zookeeper.ZKAssign(721): regionserver:43802-0x13de46b5f8b000f Attempting to transition node 55884b67b9203a71bdf8c0e3de70452d from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING 2013-04-07 12:14:50,470 DEBUG [MASTER_TABLE_OPERATIONS-asf001.sp2.ygridcore.net,33796,1365336878036-0] master.AssignmentManager(1600): Bulk assigning done for asf001.sp2.ygridcore.net,43802,1365336889418 2013-04-07 12:14:50,470 DEBUG [MASTER_TABLE_OPERATIONS-asf001.sp2.ygridcore.net,33796,1365336878036-0] master.AssignmentManager(1444): Bulk assigning 3 region(s) to asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:14:50,470 INFO [MASTER_TABLE_OPERATIONS-asf001.sp2.ygridcore.net,33796,1365336878036-0] master.RegionStates(264): Region {NAME => 'testLogSplittingAfterMasterRecoveryDueToZKExpiry,,1365336889784.fb4182aef4ce07f011871ae0a083aee0.', STARTKEY => '', ENDKEY => '1', ENCODED => fb4182aef4ce07f011871ae0a083aee0,} transitioned from {testLogSplittingAfterMasterRecoveryDueToZKExpiry,,1365336889784.fb4182aef4ce07f011871ae0a083aee0. state=OFFLINE, ts=1365336890422, server=null} to {testLogSplittingAfterMasterRecoveryDueToZKExpiry,,1365336889784.fb4182aef4ce07f011871ae0a083aee0. state=OFFLINE, ts=1365336890470, server=null} 2013-04-07 12:14:50,471 DEBUG [MASTER_TABLE_OPERATIONS-asf001.sp2.ygridcore.net,33796,1365336878036-0] zookeeper.ZKAssign(177): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Async create of unassigned node for fb4182aef4ce07f011871ae0a083aee0 with OFFLINE state 2013-04-07 12:14:50,471 INFO [MASTER_TABLE_OPERATIONS-asf001.sp2.ygridcore.net,33796,1365336878036-0] master.RegionStates(264): Region {NAME => 'testLogSplittingAfterMasterRecoveryDueToZKExpiry,3,1365336889784.4e9a3d63237d29e6bf55bf2360834cbe.', STARTKEY => '3', ENDKEY => '4', ENCODED => 4e9a3d63237d29e6bf55bf2360834cbe,} transitioned from {testLogSplittingAfterMasterRecoveryDueToZKExpiry,3,1365336889784.4e9a3d63237d29e6bf55bf2360834cbe. state=OFFLINE, ts=1365336890422, server=null} to {testLogSplittingAfterMasterRecoveryDueToZKExpiry,3,1365336889784.4e9a3d63237d29e6bf55bf2360834cbe. state=OFFLINE, ts=1365336890471, server=null} 2013-04-07 12:14:50,471 DEBUG [MASTER_TABLE_OPERATIONS-asf001.sp2.ygridcore.net,33796,1365336878036-0] zookeeper.ZKAssign(177): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Async create of unassigned node for 4e9a3d63237d29e6bf55bf2360834cbe with OFFLINE state 2013-04-07 12:14:50,471 INFO [MASTER_TABLE_OPERATIONS-asf001.sp2.ygridcore.net,33796,1365336878036-0] master.RegionStates(264): Region {NAME => 'testLogSplittingAfterMasterRecoveryDueToZKExpiry,5,1365336889784.5403a557a6168e5f3c5141eb494f6088.', STARTKEY => '5', ENDKEY => '', ENCODED => 5403a557a6168e5f3c5141eb494f6088,} transitioned from {testLogSplittingAfterMasterRecoveryDueToZKExpiry,5,1365336889784.5403a557a6168e5f3c5141eb494f6088. state=OFFLINE, ts=1365336890423, server=null} to {testLogSplittingAfterMasterRecoveryDueToZKExpiry,5,1365336889784.5403a557a6168e5f3c5141eb494f6088. state=OFFLINE, ts=1365336890471, server=null} 2013-04-07 12:14:50,471 DEBUG [MASTER_TABLE_OPERATIONS-asf001.sp2.ygridcore.net,33796,1365336878036-0] zookeeper.ZKAssign(177): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Async create of unassigned node for 5403a557a6168e5f3c5141eb494f6088 with OFFLINE state 2013-04-07 12:14:50,473 DEBUG [RS_OPEN_REGION-asf001.sp2.ygridcore.net,43802,1365336889418-0] zookeeper.ZKUtil(1680): regionserver:43802-0x13de46b5f8b000f Retrieved 152 byte(s) of data from znode /hbase/region-in-transition/567d8ef20f90f3c1b8a889d22a1c3948; data=region=testLogSplittingAfterMasterRecoveryDueToZKExpiry,4,1365336889784.567d8ef20f90f3c1b8a889d22a1c3948., state=M_ZK_REGION_OFFLINE, servername=asf001.sp2.ygridcore.net,43802,1365336889418, createTime=1365336890423, payload.length=0 2013-04-07 12:14:50,475 DEBUG [RS_OPEN_REGION-asf001.sp2.ygridcore.net,43802,1365336889418-1] zookeeper.ZKUtil(1680): regionserver:43802-0x13de46b5f8b000f Retrieved 152 byte(s) of data from znode /hbase/region-in-transition/43207168ac66dce85cf2a4f916007450; data=region=testLogSplittingAfterMasterRecoveryDueToZKExpiry,1,1365336889784.43207168ac66dce85cf2a4f916007450., state=M_ZK_REGION_OFFLINE, servername=asf001.sp2.ygridcore.net,43802,1365336889418, createTime=1365336890424, payload.length=0 2013-04-07 12:14:50,477 DEBUG [RS_OPEN_REGION-asf001.sp2.ygridcore.net,43802,1365336889418-2] zookeeper.ZKUtil(1680): regionserver:43802-0x13de46b5f8b000f Retrieved 152 byte(s) of data from znode /hbase/region-in-transition/55884b67b9203a71bdf8c0e3de70452d; data=region=testLogSplittingAfterMasterRecoveryDueToZKExpiry,2,1365336889784.55884b67b9203a71bdf8c0e3de70452d., state=M_ZK_REGION_OFFLINE, servername=asf001.sp2.ygridcore.net,43802,1365336889418, createTime=1365336890424, payload.length=0 2013-04-07 12:14:50,565 DEBUG [RS_OPEN_REGION-asf001.sp2.ygridcore.net,43802,1365336889418-0] zookeeper.ZKAssign(784): regionserver:43802-0x13de46b5f8b000f Successfully transitioned node 567d8ef20f90f3c1b8a889d22a1c3948 from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING 2013-04-07 12:14:50,565 INFO [RS_OPEN_REGION-asf001.sp2.ygridcore.net,43802,1365336889418-0] regionserver.HRegion(4023): HRegion.openHRegion Region name ==testLogSplittingAfterMasterRecoveryDueToZKExpiry,4,1365336889784.567d8ef20f90f3c1b8a889d22a1c3948. 2013-04-07 12:14:50,565 DEBUG [RS_OPEN_REGION-asf001.sp2.ygridcore.net,43802,1365336889418-0] regionserver.HRegion(4025): Opening region: {NAME => 'testLogSplittingAfterMasterRecoveryDueToZKExpiry,4,1365336889784.567d8ef20f90f3c1b8a889d22a1c3948.', STARTKEY => '4', ENDKEY => '5', ENCODED => 567d8ef20f90f3c1b8a889d22a1c3948,} 2013-04-07 12:14:50,566 DEBUG [RS_OPEN_REGION-asf001.sp2.ygridcore.net,43802,1365336889418-1] zookeeper.ZKAssign(784): regionserver:43802-0x13de46b5f8b000f Successfully transitioned node 43207168ac66dce85cf2a4f916007450 from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING 2013-04-07 12:14:50,566 INFO [RS_OPEN_REGION-asf001.sp2.ygridcore.net,43802,1365336889418-1] regionserver.HRegion(4023): HRegion.openHRegion Region name ==testLogSplittingAfterMasterRecoveryDueToZKExpiry,1,1365336889784.43207168ac66dce85cf2a4f916007450. 2013-04-07 12:14:50,566 DEBUG [RS_OPEN_REGION-asf001.sp2.ygridcore.net,43802,1365336889418-1] regionserver.HRegion(4025): Opening region: {NAME => 'testLogSplittingAfterMasterRecoveryDueToZKExpiry,1,1365336889784.43207168ac66dce85cf2a4f916007450.', STARTKEY => '1', ENDKEY => '2', ENCODED => 43207168ac66dce85cf2a4f916007450,} 2013-04-07 12:14:50,567 DEBUG [RS_OPEN_REGION-asf001.sp2.ygridcore.net,43802,1365336889418-0] regionserver.MetricsRegionSourceImpl(54): Creating new MetricsRegionSourceImpl for table testLogSplittingAfterMasterRecoveryDueToZKExpiry 567d8ef20f90f3c1b8a889d22a1c3948 2013-04-07 12:14:50,567 DEBUG [RS_OPEN_REGION-asf001.sp2.ygridcore.net,43802,1365336889418-2] zookeeper.ZKAssign(784): regionserver:43802-0x13de46b5f8b000f Successfully transitioned node 55884b67b9203a71bdf8c0e3de70452d from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING 2013-04-07 12:14:50,567 DEBUG [RS_OPEN_REGION-asf001.sp2.ygridcore.net,43802,1365336889418-0] regionserver.HRegion(496): Instantiated testLogSplittingAfterMasterRecoveryDueToZKExpiry,4,1365336889784.567d8ef20f90f3c1b8a889d22a1c3948. 2013-04-07 12:14:50,568 INFO [RS_OPEN_REGION-asf001.sp2.ygridcore.net,43802,1365336889418-2] regionserver.HRegion(4023): HRegion.openHRegion Region name ==testLogSplittingAfterMasterRecoveryDueToZKExpiry,2,1365336889784.55884b67b9203a71bdf8c0e3de70452d. 2013-04-07 12:14:50,568 DEBUG [RS_OPEN_REGION-asf001.sp2.ygridcore.net,43802,1365336889418-2] regionserver.HRegion(4025): Opening region: {NAME => 'testLogSplittingAfterMasterRecoveryDueToZKExpiry,2,1365336889784.55884b67b9203a71bdf8c0e3de70452d.', STARTKEY => '2', ENDKEY => '3', ENCODED => 55884b67b9203a71bdf8c0e3de70452d,} 2013-04-07 12:14:50,568 INFO [MASTER_TABLE_OPERATIONS-asf001.sp2.ygridcore.net,33796,1365336878036-0] master.AssignmentManager(1482): asf001.sp2.ygridcore.net,60884,1365336878389 unassigned znodes=2 of total=3 2013-04-07 12:14:50,568 DEBUG [RS_OPEN_REGION-asf001.sp2.ygridcore.net,43802,1365336889418-1] regionserver.MetricsRegionSourceImpl(54): Creating new MetricsRegionSourceImpl for table testLogSplittingAfterMasterRecoveryDueToZKExpiry 43207168ac66dce85cf2a4f916007450 2013-04-07 12:14:50,568 DEBUG [RS_OPEN_REGION-asf001.sp2.ygridcore.net,43802,1365336889418-1] regionserver.HRegion(496): Instantiated testLogSplittingAfterMasterRecoveryDueToZKExpiry,1,1365336889784.43207168ac66dce85cf2a4f916007450. 2013-04-07 12:14:50,570 DEBUG [RS_OPEN_REGION-asf001.sp2.ygridcore.net,43802,1365336889418-2] regionserver.MetricsRegionSourceImpl(54): Creating new MetricsRegionSourceImpl for table testLogSplittingAfterMasterRecoveryDueToZKExpiry 55884b67b9203a71bdf8c0e3de70452d 2013-04-07 12:14:50,570 DEBUG [hbase-am-zkevent-worker-pool-2-thread-5] master.AssignmentManager(740): Handling transition=RS_ZK_REGION_OPENING, server=asf001.sp2.ygridcore.net,43802,1365336889418, region=567d8ef20f90f3c1b8a889d22a1c3948, current state from region state map =null 2013-04-07 12:14:50,570 DEBUG [RS_OPEN_REGION-asf001.sp2.ygridcore.net,43802,1365336889418-2] regionserver.HRegion(496): Instantiated testLogSplittingAfterMasterRecoveryDueToZKExpiry,2,1365336889784.55884b67b9203a71bdf8c0e3de70452d. 2013-04-07 12:14:50,571 DEBUG [hbase-am-zkevent-worker-pool-2-thread-4] zookeeper.ZKUtil(1680): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Retrieved 152 byte(s) of data from znode /hbase/region-in-transition/5403a557a6168e5f3c5141eb494f6088 and set watcher; region=testLogSplittingAfterMasterRecoveryDueToZKExpiry,5,1365336889784.5403a557a6168e5f3c5141eb494f6088., state=M_ZK_REGION_OFFLINE, servername=asf001.sp2.ygridcore.net,60884,1365336878389, createTime=1365336890472, payload.length=0 2013-04-07 12:14:50,583 INFO [StoreOpenerThread-testLogSplittingAfterMasterRecoveryDueToZKExpiry,4,1365336889784.567d8ef20f90f3c1b8a889d22a1c3948.-1] compactions.CompactionConfiguration(85): Compaction configuration size [134217728, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; delete expired; major period 86400000, major jitter 0.200000 2013-04-07 12:14:50,584 DEBUG [MASTER_TABLE_OPERATIONS-asf001.sp2.ygridcore.net,33796,1365336878036-0] master.AssignmentManager(1600): Bulk assigning done for asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:14:50,584 DEBUG [RS_OPEN_REGION-asf001.sp2.ygridcore.net,60884,1365336878389-2] zookeeper.ZKAssign(721): regionserver:60884-0x13de46b5f8b0001 Attempting to transition node fb4182aef4ce07f011871ae0a083aee0 from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING 2013-04-07 12:14:50,584 DEBUG [RS_OPEN_REGION-asf001.sp2.ygridcore.net,60884,1365336878389-1] zookeeper.ZKAssign(721): regionserver:60884-0x13de46b5f8b0001 Attempting to transition node 5403a557a6168e5f3c5141eb494f6088 from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING 2013-04-07 12:14:50,585 DEBUG [hbase-am-zkevent-worker-pool-2-thread-7] zookeeper.ZKUtil(1680): hconnection-0x1f327e-0x13de46b5f8b0003 Retrieved 43 byte(s) of data from znode /hbase/meta-region-server; data=asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:14:50,587 DEBUG [RS_OPEN_REGION-asf001.sp2.ygridcore.net,60884,1365336878389-0] zookeeper.ZKUtil(1680): regionserver:60884-0x13de46b5f8b0001 Retrieved 152 byte(s) of data from znode /hbase/region-in-transition/4e9a3d63237d29e6bf55bf2360834cbe; data=region=testLogSplittingAfterMasterRecoveryDueToZKExpiry,3,1365336889784.4e9a3d63237d29e6bf55bf2360834cbe., state=M_ZK_REGION_OFFLINE, servername=asf001.sp2.ygridcore.net,60884,1365336878389, createTime=1365336890471, payload.length=0 2013-04-07 12:14:50,588 INFO [hbase-am-zkevent-worker-pool-2-thread-7] master.RegionStates(264): Region {NAME => 'testLogSplittingAfterMasterRecoveryDueToZKExpiry,2,1365336889784.55884b67b9203a71bdf8c0e3de70452d.', STARTKEY => '2', ENDKEY => '3', ENCODED => 55884b67b9203a71bdf8c0e3de70452d,} transitioned from {testLogSplittingAfterMasterRecoveryDueToZKExpiry,2,1365336889784.55884b67b9203a71bdf8c0e3de70452d. state=OFFLINE, ts=1365336890587, server=null} to {testLogSplittingAfterMasterRecoveryDueToZKExpiry,2,1365336889784.55884b67b9203a71bdf8c0e3de70452d. state=OPENING, ts=1365336890587, server=asf001.sp2.ygridcore.net,43802,1365336889418} 2013-04-07 12:14:50,589 DEBUG [hbase-am-zkevent-worker-pool-2-thread-6] zookeeper.ZKUtil(1680): hconnection-0x1f327e-0x13de46b5f8b0003 Retrieved 43 byte(s) of data from znode /hbase/meta-region-server; data=asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:14:50,590 DEBUG [RS_OPEN_REGION-asf001.sp2.ygridcore.net,60884,1365336878389-2] zookeeper.ZKUtil(1680): regionserver:60884-0x13de46b5f8b0001 Retrieved 151 byte(s) of data from znode /hbase/region-in-transition/fb4182aef4ce07f011871ae0a083aee0; data=region=testLogSplittingAfterMasterRecoveryDueToZKExpiry,,1365336889784.fb4182aef4ce07f011871ae0a083aee0., state=M_ZK_REGION_OFFLINE, servername=asf001.sp2.ygridcore.net,60884,1365336878389, createTime=1365336890471, payload.length=0 2013-04-07 12:14:50,592 DEBUG [RS_OPEN_REGION-asf001.sp2.ygridcore.net,60884,1365336878389-1] zookeeper.ZKUtil(1680): regionserver:60884-0x13de46b5f8b0001 Retrieved 152 byte(s) of data from znode /hbase/region-in-transition/5403a557a6168e5f3c5141eb494f6088; data=region=testLogSplittingAfterMasterRecoveryDueToZKExpiry,5,1365336889784.5403a557a6168e5f3c5141eb494f6088., state=M_ZK_REGION_OFFLINE, servername=asf001.sp2.ygridcore.net,60884,1365336878389, createTime=1365336890472, payload.length=0 2013-04-07 12:14:50,602 INFO [StoreOpenerThread-testLogSplittingAfterMasterRecoveryDueToZKExpiry,1,1365336889784.43207168ac66dce85cf2a4f916007450.-1] compactions.CompactionConfiguration(85): Compaction configuration size [134217728, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; delete expired; major period 86400000, major jitter 0.200000 2013-04-07 12:14:50,603 DEBUG [RS_OPEN_REGION-asf001.sp2.ygridcore.net,60884,1365336878389-0] regionserver.MetricsRegionSourceImpl(54): Creating new MetricsRegionSourceImpl for table testLogSplittingAfterMasterRecoveryDueToZKExpiry 4e9a3d63237d29e6bf55bf2360834cbe 2013-04-07 12:14:50,603 DEBUG [RS_OPEN_REGION-asf001.sp2.ygridcore.net,60884,1365336878389-0] regionserver.HRegion(496): Instantiated testLogSplittingAfterMasterRecoveryDueToZKExpiry,3,1365336889784.4e9a3d63237d29e6bf55bf2360834cbe. 2013-04-07 12:14:50,605 DEBUG [RS_OPEN_REGION-asf001.sp2.ygridcore.net,60884,1365336878389-2] zookeeper.ZKAssign(784): regionserver:60884-0x13de46b5f8b0001 Successfully transitioned node fb4182aef4ce07f011871ae0a083aee0 from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING 2013-04-07 12:14:50,605 INFO [RS_OPEN_REGION-asf001.sp2.ygridcore.net,60884,1365336878389-2] regionserver.HRegion(4023): HRegion.openHRegion Region name ==testLogSplittingAfterMasterRecoveryDueToZKExpiry,,1365336889784.fb4182aef4ce07f011871ae0a083aee0. 2013-04-07 12:14:50,605 DEBUG [RS_OPEN_REGION-asf001.sp2.ygridcore.net,60884,1365336878389-2] regionserver.HRegion(4025): Opening region: {NAME => 'testLogSplittingAfterMasterRecoveryDueToZKExpiry,,1365336889784.fb4182aef4ce07f011871ae0a083aee0.', STARTKEY => '', ENDKEY => '1', ENCODED => fb4182aef4ce07f011871ae0a083aee0,} 2013-04-07 12:14:50,605 DEBUG [RS_OPEN_REGION-asf001.sp2.ygridcore.net,60884,1365336878389-1] zookeeper.ZKAssign(784): regionserver:60884-0x13de46b5f8b0001 Successfully transitioned node 5403a557a6168e5f3c5141eb494f6088 from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING 2013-04-07 12:14:50,606 DEBUG [hbase-am-zkevent-worker-pool-2-thread-6] zookeeper.ZKUtil(1680): hconnection-0x1f327e-0x13de46b5f8b0003 Retrieved 43 byte(s) of data from znode /hbase/meta-region-server; data=asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:14:50,606 INFO [RS_OPEN_REGION-asf001.sp2.ygridcore.net,43802,1365336889418-0] regionserver.HRegion(586): Onlined testLogSplittingAfterMasterRecoveryDueToZKExpiry,4,1365336889784.567d8ef20f90f3c1b8a889d22a1c3948.; next sequenceid=1 2013-04-07 12:14:50,606 INFO [RS_OPEN_REGION-asf001.sp2.ygridcore.net,60884,1365336878389-1] regionserver.HRegion(4023): HRegion.openHRegion Region name ==testLogSplittingAfterMasterRecoveryDueToZKExpiry,5,1365336889784.5403a557a6168e5f3c5141eb494f6088. 2013-04-07 12:14:50,606 DEBUG [RS_OPEN_REGION-asf001.sp2.ygridcore.net,43802,1365336889418-0] zookeeper.ZKAssign(721): regionserver:43802-0x13de46b5f8b000f Attempting to transition node 567d8ef20f90f3c1b8a889d22a1c3948 from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENING 2013-04-07 12:14:50,606 DEBUG [RS_OPEN_REGION-asf001.sp2.ygridcore.net,60884,1365336878389-1] regionserver.HRegion(4025): Opening region: {NAME => 'testLogSplittingAfterMasterRecoveryDueToZKExpiry,5,1365336889784.5403a557a6168e5f3c5141eb494f6088.', STARTKEY => '5', ENDKEY => '', ENCODED => 5403a557a6168e5f3c5141eb494f6088,} 2013-04-07 12:14:50,606 DEBUG [MASTER_TABLE_OPERATIONS-asf001.sp2.ygridcore.net,33796,1365336878036-0] master.TableLockManager$ZKTableLockManager$TableLockImpl(253): Attempt to release table write lock on :testLogSplittingAfterMasterRecoveryDueToZKExpiry 2013-04-07 12:14:50,606 DEBUG [hbase-am-zkevent-worker-pool-2-thread-4] zookeeper.ZKUtil(1680): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Retrieved 152 byte(s) of data from znode /hbase/region-in-transition/4e9a3d63237d29e6bf55bf2360834cbe and set watcher; region=testLogSplittingAfterMasterRecoveryDueToZKExpiry,3,1365336889784.4e9a3d63237d29e6bf55bf2360834cbe., state=RS_ZK_REGION_OPENING, servername=asf001.sp2.ygridcore.net,60884,1365336878389, createTime=1365336890587, payload.length=0 2013-04-07 12:14:50,608 INFO [hbase-am-zkevent-worker-pool-2-thread-6] master.RegionStates(264): Region {NAME => 'testLogSplittingAfterMasterRecoveryDueToZKExpiry,1,1365336889784.43207168ac66dce85cf2a4f916007450.', STARTKEY => '1', ENDKEY => '2', ENCODED => 43207168ac66dce85cf2a4f916007450,} transitioned from {testLogSplittingAfterMasterRecoveryDueToZKExpiry,1,1365336889784.43207168ac66dce85cf2a4f916007450. state=OFFLINE, ts=1365336890608, server=null} to {testLogSplittingAfterMasterRecoveryDueToZKExpiry,1,1365336889784.43207168ac66dce85cf2a4f916007450. state=OPENING, ts=1365336890608, server=asf001.sp2.ygridcore.net,43802,1365336889418} 2013-04-07 12:14:50,608 INFO [StoreOpenerThread-testLogSplittingAfterMasterRecoveryDueToZKExpiry,2,1365336889784.55884b67b9203a71bdf8c0e3de70452d.-1] compactions.CompactionConfiguration(85): Compaction configuration size [134217728, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; delete expired; major period 86400000, major jitter 0.200000 2013-04-07 12:14:50,609 DEBUG [RS_OPEN_REGION-asf001.sp2.ygridcore.net,60884,1365336878389-2] regionserver.MetricsRegionSourceImpl(54): Creating new MetricsRegionSourceImpl for table testLogSplittingAfterMasterRecoveryDueToZKExpiry fb4182aef4ce07f011871ae0a083aee0 2013-04-07 12:14:50,609 DEBUG [RS_OPEN_REGION-asf001.sp2.ygridcore.net,60884,1365336878389-2] regionserver.HRegion(496): Instantiated testLogSplittingAfterMasterRecoveryDueToZKExpiry,,1365336889784.fb4182aef4ce07f011871ae0a083aee0. 2013-04-07 12:14:50,609 DEBUG [RS_OPEN_REGION-asf001.sp2.ygridcore.net,43802,1365336889418-0] zookeeper.ZKUtil(1680): regionserver:43802-0x13de46b5f8b000f Retrieved 152 byte(s) of data from znode /hbase/region-in-transition/567d8ef20f90f3c1b8a889d22a1c3948; data=region=testLogSplittingAfterMasterRecoveryDueToZKExpiry,4,1365336889784.567d8ef20f90f3c1b8a889d22a1c3948., state=RS_ZK_REGION_OPENING, servername=asf001.sp2.ygridcore.net,43802,1365336889418, createTime=1365336890473, payload.length=0 2013-04-07 12:14:50,610 DEBUG [RS_OPEN_REGION-asf001.sp2.ygridcore.net,60884,1365336878389-1] regionserver.MetricsRegionSourceImpl(54): Creating new MetricsRegionSourceImpl for table testLogSplittingAfterMasterRecoveryDueToZKExpiry 5403a557a6168e5f3c5141eb494f6088 2013-04-07 12:14:50,610 DEBUG [RS_OPEN_REGION-asf001.sp2.ygridcore.net,60884,1365336878389-1] regionserver.HRegion(496): Instantiated testLogSplittingAfterMasterRecoveryDueToZKExpiry,5,1365336889784.5403a557a6168e5f3c5141eb494f6088. 2013-04-07 12:14:50,611 INFO [RS_OPEN_REGION-asf001.sp2.ygridcore.net,43802,1365336889418-1] regionserver.HRegion(586): Onlined testLogSplittingAfterMasterRecoveryDueToZKExpiry,1,1365336889784.43207168ac66dce85cf2a4f916007450.; next sequenceid=1 2013-04-07 12:14:50,611 DEBUG [RS_OPEN_REGION-asf001.sp2.ygridcore.net,43802,1365336889418-1] zookeeper.ZKAssign(721): regionserver:43802-0x13de46b5f8b000f Attempting to transition node 43207168ac66dce85cf2a4f916007450 from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENING 2013-04-07 12:14:50,612 INFO [StoreOpenerThread-testLogSplittingAfterMasterRecoveryDueToZKExpiry,3,1365336889784.4e9a3d63237d29e6bf55bf2360834cbe.-1] compactions.CompactionConfiguration(85): Compaction configuration size [134217728, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; delete expired; major period 86400000, major jitter 0.200000 2013-04-07 12:14:50,615 DEBUG [RS_OPEN_REGION-asf001.sp2.ygridcore.net,43802,1365336889418-0] zookeeper.ZKAssign(784): regionserver:43802-0x13de46b5f8b000f Successfully transitioned node 567d8ef20f90f3c1b8a889d22a1c3948 from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENING 2013-04-07 12:14:50,615 INFO [PostOpenDeployTasks:567d8ef20f90f3c1b8a889d22a1c3948] regionserver.HRegionServer(1563): Post open deploy tasks for region=testLogSplittingAfterMasterRecoveryDueToZKExpiry,4,1365336889784.567d8ef20f90f3c1b8a889d22a1c3948. 2013-04-07 12:14:50,616 DEBUG [hbase-am-zkevent-worker-pool-2-thread-8] zookeeper.ZKUtil(1680): hconnection-0x1f327e-0x13de46b5f8b0003 Retrieved 43 byte(s) of data from znode /hbase/meta-region-server; data=asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:14:50,618 DEBUG [RS_OPEN_REGION-asf001.sp2.ygridcore.net,43802,1365336889418-1] zookeeper.ZKUtil(1680): regionserver:43802-0x13de46b5f8b000f Retrieved 152 byte(s) of data from znode /hbase/region-in-transition/43207168ac66dce85cf2a4f916007450; data=region=testLogSplittingAfterMasterRecoveryDueToZKExpiry,1,1365336889784.43207168ac66dce85cf2a4f916007450., state=RS_ZK_REGION_OPENING, servername=asf001.sp2.ygridcore.net,43802,1365336889418, createTime=1365336890475, payload.length=0 2013-04-07 12:14:50,620 DEBUG [PostOpenDeployTasks:567d8ef20f90f3c1b8a889d22a1c3948] zookeeper.ZKUtil(1680): hconnection-0x1c9f8fb-0x13de46b5f8b0010 Retrieved 43 byte(s) of data from znode /hbase/meta-region-server; data=asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:14:50,620 DEBUG [hbase-am-zkevent-worker-pool-2-thread-8] zookeeper.ZKUtil(1680): hconnection-0x1f327e-0x13de46b5f8b0003 Retrieved 43 byte(s) of data from znode /hbase/meta-region-server; data=asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:14:50,620 DEBUG [hbase-am-zkevent-worker-pool-2-thread-10] zookeeper.ZKUtil(1680): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Retrieved 152 byte(s) of data from znode /hbase/region-in-transition/5403a557a6168e5f3c5141eb494f6088 and set watcher; region=testLogSplittingAfterMasterRecoveryDueToZKExpiry,5,1365336889784.5403a557a6168e5f3c5141eb494f6088., state=RS_ZK_REGION_OPENING, servername=asf001.sp2.ygridcore.net,60884,1365336878389, createTime=1365336890592, payload.length=0 2013-04-07 12:14:50,629 DEBUG [PostOpenDeployTasks:567d8ef20f90f3c1b8a889d22a1c3948] zookeeper.ZKUtil(1680): hconnection-0x1c9f8fb-0x13de46b5f8b0010 Retrieved 43 byte(s) of data from znode /hbase/meta-region-server; data=asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:14:50,629 DEBUG [hbase-am-zkevent-worker-pool-2-thread-10] master.AssignmentManager(740): Handling transition=RS_ZK_REGION_OPENING, server=asf001.sp2.ygridcore.net,60884,1365336878389, region=5403a557a6168e5f3c5141eb494f6088, current state from region state map =null 2013-04-07 12:14:50,629 DEBUG [RS_OPEN_REGION-asf001.sp2.ygridcore.net,43802,1365336889418-1] zookeeper.ZKAssign(784): regionserver:43802-0x13de46b5f8b000f Successfully transitioned node 43207168ac66dce85cf2a4f916007450 from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENING 2013-04-07 12:14:50,629 DEBUG [hbase-am-zkevent-worker-pool-2-thread-9] master.AssignmentManager(740): Handling transition=RS_ZK_REGION_OPENING, server=asf001.sp2.ygridcore.net,60884,1365336878389, region=fb4182aef4ce07f011871ae0a083aee0, current state from region state map =null 2013-04-07 12:14:50,631 INFO [PostOpenDeployTasks:43207168ac66dce85cf2a4f916007450] regionserver.HRegionServer(1563): Post open deploy tasks for region=testLogSplittingAfterMasterRecoveryDueToZKExpiry,1,1365336889784.43207168ac66dce85cf2a4f916007450. 2013-04-07 12:14:50,635 DEBUG [hbase-am-zkevent-worker-pool-2-thread-10] zookeeper.ZKUtil(1680): hconnection-0x1f327e-0x13de46b5f8b0003 Retrieved 43 byte(s) of data from znode /hbase/meta-region-server; data=asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:14:50,637 DEBUG [hbase-am-zkevent-worker-pool-2-thread-12] zookeeper.ZKUtil(1680): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Retrieved 152 byte(s) of data from znode /hbase/region-in-transition/43207168ac66dce85cf2a4f916007450 and set watcher; region=testLogSplittingAfterMasterRecoveryDueToZKExpiry,1,1365336889784.43207168ac66dce85cf2a4f916007450., state=RS_ZK_REGION_OPENING, servername=asf001.sp2.ygridcore.net,43802,1365336889418, createTime=1365336890618, payload.length=0 2013-04-07 12:14:50,637 DEBUG [RS_OPEN_REGION-asf001.sp2.ygridcore.net,60884,1365336878389-0] zookeeper.ZKUtil(1680): regionserver:60884-0x13de46b5f8b0001 Retrieved 152 byte(s) of data from znode /hbase/region-in-transition/4e9a3d63237d29e6bf55bf2360834cbe; data=region=testLogSplittingAfterMasterRecoveryDueToZKExpiry,3,1365336889784.4e9a3d63237d29e6bf55bf2360834cbe., state=RS_ZK_REGION_OPENING, servername=asf001.sp2.ygridcore.net,60884,1365336878389, createTime=1365336890587, payload.length=0 2013-04-07 12:14:50,638 DEBUG [PostOpenDeployTasks:43207168ac66dce85cf2a4f916007450] zookeeper.ZKUtil(1680): hconnection-0x1c9f8fb-0x13de46b5f8b0010 Retrieved 43 byte(s) of data from znode /hbase/meta-region-server; data=asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:14:50,639 INFO [PostOpenDeployTasks:567d8ef20f90f3c1b8a889d22a1c3948] catalog.MetaEditor(432): Updated row testLogSplittingAfterMasterRecoveryDueToZKExpiry,4,1365336889784.567d8ef20f90f3c1b8a889d22a1c3948. with server=asf001.sp2.ygridcore.net,43802,1365336889418 2013-04-07 12:14:50,639 INFO [PostOpenDeployTasks:567d8ef20f90f3c1b8a889d22a1c3948] regionserver.HRegionServer(1584): Done with post open deploy task for region=testLogSplittingAfterMasterRecoveryDueToZKExpiry,4,1365336889784.567d8ef20f90f3c1b8a889d22a1c3948. 2013-04-07 12:14:50,640 DEBUG [RS_OPEN_REGION-asf001.sp2.ygridcore.net,43802,1365336889418-0] zookeeper.ZKAssign(721): regionserver:43802-0x13de46b5f8b000f Attempting to transition node 567d8ef20f90f3c1b8a889d22a1c3948 from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENED 2013-04-07 12:14:50,640 INFO [hbase-am-zkevent-worker-pool-2-thread-10] master.RegionStates(264): Region {NAME => 'testLogSplittingAfterMasterRecoveryDueToZKExpiry,5,1365336889784.5403a557a6168e5f3c5141eb494f6088.', STARTKEY => '5', ENDKEY => '', ENCODED => 5403a557a6168e5f3c5141eb494f6088,} transitioned from {testLogSplittingAfterMasterRecoveryDueToZKExpiry,5,1365336889784.5403a557a6168e5f3c5141eb494f6088. state=OFFLINE, ts=1365336890640, server=null} to {testLogSplittingAfterMasterRecoveryDueToZKExpiry,5,1365336889784.5403a557a6168e5f3c5141eb494f6088. state=OPENING, ts=1365336890640, server=asf001.sp2.ygridcore.net,60884,1365336878389} 2013-04-07 12:14:50,640 DEBUG [hbase-am-zkevent-worker-pool-2-thread-12] master.AssignmentManager(740): Handling transition=RS_ZK_REGION_OPENING, server=asf001.sp2.ygridcore.net,43802,1365336889418, region=43207168ac66dce85cf2a4f916007450, current state from region state map ={testLogSplittingAfterMasterRecoveryDueToZKExpiry,1,1365336889784.43207168ac66dce85cf2a4f916007450. state=OPENING, ts=1365336890608, server=asf001.sp2.ygridcore.net,43802,1365336889418} 2013-04-07 12:14:50,643 DEBUG [PostOpenDeployTasks:43207168ac66dce85cf2a4f916007450] zookeeper.ZKUtil(1680): hconnection-0x1c9f8fb-0x13de46b5f8b0010 Retrieved 43 byte(s) of data from znode /hbase/meta-region-server; data=asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:14:50,644 DEBUG [RS_OPEN_REGION-asf001.sp2.ygridcore.net,60884,1365336878389-0] zookeeper.ZKAssign(784): regionserver:60884-0x13de46b5f8b0001 Successfully transitioned node 4e9a3d63237d29e6bf55bf2360834cbe from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENING 2013-04-07 12:14:50,644 DEBUG [RS_OPEN_REGION-asf001.sp2.ygridcore.net,43802,1365336889418-0] zookeeper.ZKUtil(1680): regionserver:43802-0x13de46b5f8b000f Retrieved 152 byte(s) of data from znode /hbase/region-in-transition/567d8ef20f90f3c1b8a889d22a1c3948; data=region=testLogSplittingAfterMasterRecoveryDueToZKExpiry,4,1365336889784.567d8ef20f90f3c1b8a889d22a1c3948., state=RS_ZK_REGION_OPENING, servername=asf001.sp2.ygridcore.net,43802,1365336889418, createTime=1365336890609, payload.length=0 2013-04-07 12:14:50,644 INFO [PostOpenDeployTasks:4e9a3d63237d29e6bf55bf2360834cbe] regionserver.HRegionServer(1563): Post open deploy tasks for region=testLogSplittingAfterMasterRecoveryDueToZKExpiry,3,1365336889784.4e9a3d63237d29e6bf55bf2360834cbe. 2013-04-07 12:14:50,644 DEBUG [hbase-am-zkevent-worker-pool-2-thread-9] zookeeper.ZKUtil(1680): hconnection-0x1f327e-0x13de46b5f8b0003 Retrieved 43 byte(s) of data from znode /hbase/meta-region-server; data=asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:14:50,645 DEBUG [RS_OPEN_REGION-asf001.sp2.ygridcore.net,60884,1365336878389-1] zookeeper.ZKUtil(1680): regionserver:60884-0x13de46b5f8b0001 Retrieved 152 byte(s) of data from znode /hbase/region-in-transition/5403a557a6168e5f3c5141eb494f6088; data=region=testLogSplittingAfterMasterRecoveryDueToZKExpiry,5,1365336889784.5403a557a6168e5f3c5141eb494f6088., state=RS_ZK_REGION_OPENING, servername=asf001.sp2.ygridcore.net,60884,1365336878389, createTime=1365336890592, payload.length=0 2013-04-07 12:14:50,647 INFO [PostOpenDeployTasks:43207168ac66dce85cf2a4f916007450] catalog.MetaEditor(432): Updated row testLogSplittingAfterMasterRecoveryDueToZKExpiry,1,1365336889784.43207168ac66dce85cf2a4f916007450. with server=asf001.sp2.ygridcore.net,43802,1365336889418 2013-04-07 12:14:50,647 INFO [PostOpenDeployTasks:43207168ac66dce85cf2a4f916007450] regionserver.HRegionServer(1584): Done with post open deploy task for region=testLogSplittingAfterMasterRecoveryDueToZKExpiry,1,1365336889784.43207168ac66dce85cf2a4f916007450. 2013-04-07 12:14:50,648 DEBUG [RS_OPEN_REGION-asf001.sp2.ygridcore.net,43802,1365336889418-1] zookeeper.ZKAssign(721): regionserver:43802-0x13de46b5f8b000f Attempting to transition node 43207168ac66dce85cf2a4f916007450 from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENED 2013-04-07 12:14:50,648 DEBUG [PostOpenDeployTasks:4e9a3d63237d29e6bf55bf2360834cbe] zookeeper.ZKUtil(1680): hconnection-0x43ad4a-0x13de46b5f8b0005 Retrieved 43 byte(s) of data from znode /hbase/meta-region-server; data=asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:14:50,648 DEBUG [hbase-am-zkevent-worker-pool-2-thread-13] zookeeper.ZKUtil(1680): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Retrieved 152 byte(s) of data from znode /hbase/region-in-transition/4e9a3d63237d29e6bf55bf2360834cbe and set watcher; region=testLogSplittingAfterMasterRecoveryDueToZKExpiry,3,1365336889784.4e9a3d63237d29e6bf55bf2360834cbe., state=RS_ZK_REGION_OPENING, servername=asf001.sp2.ygridcore.net,60884,1365336878389, createTime=1365336890637, payload.length=0 2013-04-07 12:14:50,649 DEBUG [RS_OPEN_REGION-asf001.sp2.ygridcore.net,43802,1365336889418-0] zookeeper.ZKAssign(784): regionserver:43802-0x13de46b5f8b000f Successfully transitioned node 567d8ef20f90f3c1b8a889d22a1c3948 from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENED 2013-04-07 12:14:50,649 DEBUG [RS_OPEN_REGION-asf001.sp2.ygridcore.net,43802,1365336889418-0] handler.OpenRegionHandler(361): region transitioned to opened in zookeeper: {NAME => 'testLogSplittingAfterMasterRecoveryDueToZKExpiry,4,1365336889784.567d8ef20f90f3c1b8a889d22a1c3948.', STARTKEY => '4', ENDKEY => '5', ENCODED => 567d8ef20f90f3c1b8a889d22a1c3948,}, server: asf001.sp2.ygridcore.net,43802,1365336889418 2013-04-07 12:14:50,649 DEBUG [RS_OPEN_REGION-asf001.sp2.ygridcore.net,43802,1365336889418-0] handler.OpenRegionHandler(168): Opened testLogSplittingAfterMasterRecoveryDueToZKExpiry,4,1365336889784.567d8ef20f90f3c1b8a889d22a1c3948. on server:asf001.sp2.ygridcore.net,43802,1365336889418 2013-04-07 12:14:50,650 INFO [RS_OPEN_REGION-asf001.sp2.ygridcore.net,43802,1365336889418-2] regionserver.HRegion(586): Onlined testLogSplittingAfterMasterRecoveryDueToZKExpiry,2,1365336889784.55884b67b9203a71bdf8c0e3de70452d.; next sequenceid=1 2013-04-07 12:14:50,651 DEBUG [RS_OPEN_REGION-asf001.sp2.ygridcore.net,43802,1365336889418-2] zookeeper.ZKAssign(721): regionserver:43802-0x13de46b5f8b000f Attempting to transition node 55884b67b9203a71bdf8c0e3de70452d from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENING 2013-04-07 12:14:50,654 DEBUG [RS_OPEN_REGION-asf001.sp2.ygridcore.net,43802,1365336889418-1] zookeeper.ZKUtil(1680): regionserver:43802-0x13de46b5f8b000f Retrieved 152 byte(s) of data from znode /hbase/region-in-transition/43207168ac66dce85cf2a4f916007450; data=region=testLogSplittingAfterMasterRecoveryDueToZKExpiry,1,1365336889784.43207168ac66dce85cf2a4f916007450., state=RS_ZK_REGION_OPENING, servername=asf001.sp2.ygridcore.net,43802,1365336889418, createTime=1365336890618, payload.length=0 2013-04-07 12:14:50,656 INFO [PostOpenDeployTasks:4e9a3d63237d29e6bf55bf2360834cbe] catalog.MetaEditor(432): Updated row testLogSplittingAfterMasterRecoveryDueToZKExpiry,3,1365336889784.4e9a3d63237d29e6bf55bf2360834cbe. with server=asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:14:50,656 INFO [PostOpenDeployTasks:4e9a3d63237d29e6bf55bf2360834cbe] regionserver.HRegionServer(1584): Done with post open deploy task for region=testLogSplittingAfterMasterRecoveryDueToZKExpiry,3,1365336889784.4e9a3d63237d29e6bf55bf2360834cbe. 2013-04-07 12:14:50,658 DEBUG [RS_OPEN_REGION-asf001.sp2.ygridcore.net,60884,1365336878389-0] zookeeper.ZKAssign(721): regionserver:60884-0x13de46b5f8b0001 Attempting to transition node 4e9a3d63237d29e6bf55bf2360834cbe from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENED 2013-04-07 12:14:50,659 DEBUG [RS_OPEN_REGION-asf001.sp2.ygridcore.net,43802,1365336889418-2] zookeeper.ZKUtil(1680): regionserver:43802-0x13de46b5f8b000f Retrieved 152 byte(s) of data from znode /hbase/region-in-transition/55884b67b9203a71bdf8c0e3de70452d; data=region=testLogSplittingAfterMasterRecoveryDueToZKExpiry,2,1365336889784.55884b67b9203a71bdf8c0e3de70452d., state=RS_ZK_REGION_OPENING, servername=asf001.sp2.ygridcore.net,43802,1365336889418, createTime=1365336890477, payload.length=0 2013-04-07 12:14:50,660 DEBUG [hbase-am-zkevent-worker-pool-2-thread-14] master.AssignmentManager(740): Handling transition=RS_ZK_REGION_OPENED, server=asf001.sp2.ygridcore.net,43802,1365336889418, region=567d8ef20f90f3c1b8a889d22a1c3948, current state from region state map ={testLogSplittingAfterMasterRecoveryDueToZKExpiry,4,1365336889784.567d8ef20f90f3c1b8a889d22a1c3948. state=OPENING, ts=1365336890628, server=asf001.sp2.ygridcore.net,43802,1365336889418} 2013-04-07 12:14:50,660 INFO [hbase-am-zkevent-worker-pool-2-thread-14] master.RegionStates(264): Region {NAME => 'testLogSplittingAfterMasterRecoveryDueToZKExpiry,4,1365336889784.567d8ef20f90f3c1b8a889d22a1c3948.', STARTKEY => '4', ENDKEY => '5', ENCODED => 567d8ef20f90f3c1b8a889d22a1c3948,} transitioned from {testLogSplittingAfterMasterRecoveryDueToZKExpiry,4,1365336889784.567d8ef20f90f3c1b8a889d22a1c3948. state=OPENING, ts=1365336890628, server=asf001.sp2.ygridcore.net,43802,1365336889418} to {testLogSplittingAfterMasterRecoveryDueToZKExpiry,4,1365336889784.567d8ef20f90f3c1b8a889d22a1c3948. state=OPEN, ts=1365336890660, server=asf001.sp2.ygridcore.net,43802,1365336889418} 2013-04-07 12:14:50,660 DEBUG [MASTER_OPEN_REGION-asf001.sp2.ygridcore.net,33796,1365336878036-1] handler.OpenedRegionHandler(146): Handling OPENED event for testLogSplittingAfterMasterRecoveryDueToZKExpiry,4,1365336889784.567d8ef20f90f3c1b8a889d22a1c3948. from asf001.sp2.ygridcore.net,43802,1365336889418; deleting unassigned node 2013-04-07 12:14:50,660 DEBUG [MASTER_OPEN_REGION-asf001.sp2.ygridcore.net,33796,1365336878036-1] zookeeper.ZKAssign(405): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Deleting existing unassigned node for 567d8ef20f90f3c1b8a889d22a1c3948 that is in expected state RS_ZK_REGION_OPENED 2013-04-07 12:14:50,661 DEBUG [PostOpenDeployTasks:5403a557a6168e5f3c5141eb494f6088] zookeeper.ZKUtil(1680): hconnection-0x43ad4a-0x13de46b5f8b0005 Retrieved 43 byte(s) of data from znode /hbase/meta-region-server; data=asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:14:50,662 DEBUG [RS_OPEN_REGION-asf001.sp2.ygridcore.net,60884,1365336878389-0] zookeeper.ZKUtil(1680): regionserver:60884-0x13de46b5f8b0001 Retrieved 152 byte(s) of data from znode /hbase/region-in-transition/4e9a3d63237d29e6bf55bf2360834cbe; data=region=testLogSplittingAfterMasterRecoveryDueToZKExpiry,3,1365336889784.4e9a3d63237d29e6bf55bf2360834cbe., state=RS_ZK_REGION_OPENING, servername=asf001.sp2.ygridcore.net,60884,1365336878389, createTime=1365336890637, payload.length=0 2013-04-07 12:14:50,662 DEBUG [RS_OPEN_REGION-asf001.sp2.ygridcore.net,43802,1365336889418-1] zookeeper.ZKAssign(784): regionserver:43802-0x13de46b5f8b000f Successfully transitioned node 43207168ac66dce85cf2a4f916007450 from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENED 2013-04-07 12:14:50,662 DEBUG [RS_OPEN_REGION-asf001.sp2.ygridcore.net,43802,1365336889418-1] handler.OpenRegionHandler(361): region transitioned to opened in zookeeper: {NAME => 'testLogSplittingAfterMasterRecoveryDueToZKExpiry,1,1365336889784.43207168ac66dce85cf2a4f916007450.', STARTKEY => '1', ENDKEY => '2', ENCODED => 43207168ac66dce85cf2a4f916007450,}, server: asf001.sp2.ygridcore.net,43802,1365336889418 2013-04-07 12:14:50,663 DEBUG [RS_OPEN_REGION-asf001.sp2.ygridcore.net,43802,1365336889418-1] handler.OpenRegionHandler(168): Opened testLogSplittingAfterMasterRecoveryDueToZKExpiry,1,1365336889784.43207168ac66dce85cf2a4f916007450. on server:asf001.sp2.ygridcore.net,43802,1365336889418 2013-04-07 12:14:50,672 DEBUG [RS_OPEN_REGION-asf001.sp2.ygridcore.net,43802,1365336889418-2] zookeeper.ZKAssign(784): regionserver:43802-0x13de46b5f8b000f Successfully transitioned node 55884b67b9203a71bdf8c0e3de70452d from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENING 2013-04-07 12:14:50,686 INFO [PostOpenDeployTasks:55884b67b9203a71bdf8c0e3de70452d] regionserver.HRegionServer(1563): Post open deploy tasks for region=testLogSplittingAfterMasterRecoveryDueToZKExpiry,2,1365336889784.55884b67b9203a71bdf8c0e3de70452d. 2013-04-07 12:14:50,686 INFO [RS_OPEN_REGION-asf001.sp2.ygridcore.net,60884,1365336878389-2] regionserver.HRegion(586): Onlined testLogSplittingAfterMasterRecoveryDueToZKExpiry,,1365336889784.fb4182aef4ce07f011871ae0a083aee0.; next sequenceid=1 2013-04-07 12:14:50,686 DEBUG [RS_OPEN_REGION-asf001.sp2.ygridcore.net,60884,1365336878389-1] zookeeper.ZKUtil(1680): regionserver:60884-0x13de46b5f8b0001 Retrieved 152 byte(s) of data from znode /hbase/region-in-transition/5403a557a6168e5f3c5141eb494f6088; data=region=testLogSplittingAfterMasterRecoveryDueToZKExpiry,5,1365336889784.5403a557a6168e5f3c5141eb494f6088., state=RS_ZK_REGION_OPENING, servername=asf001.sp2.ygridcore.net,60884,1365336878389, createTime=1365336890646, payload.length=0 2013-04-07 12:14:50,687 DEBUG [RS_OPEN_REGION-asf001.sp2.ygridcore.net,60884,1365336878389-2] zookeeper.ZKAssign(721): regionserver:60884-0x13de46b5f8b0001 Attempting to transition node fb4182aef4ce07f011871ae0a083aee0 from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENING 2013-04-07 12:14:50,688 DEBUG [PostOpenDeployTasks:55884b67b9203a71bdf8c0e3de70452d] zookeeper.ZKUtil(1680): hconnection-0x1c9f8fb-0x13de46b5f8b0010 Retrieved 43 byte(s) of data from znode /hbase/meta-region-server; data=asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:14:50,689 DEBUG [PostOpenDeployTasks:55884b67b9203a71bdf8c0e3de70452d] zookeeper.ZKUtil(1680): hconnection-0x1c9f8fb-0x13de46b5f8b0010 Retrieved 43 byte(s) of data from znode /hbase/meta-region-server; data=asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:14:50,692 DEBUG [RS_OPEN_REGION-asf001.sp2.ygridcore.net,60884,1365336878389-1] zookeeper.ZKAssign(784): regionserver:60884-0x13de46b5f8b0001 Successfully transitioned node 5403a557a6168e5f3c5141eb494f6088 from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENED 2013-04-07 12:14:50,692 DEBUG [RS_OPEN_REGION-asf001.sp2.ygridcore.net,60884,1365336878389-1] handler.OpenRegionHandler(361): region transitioned to opened in zookeeper: {NAME => 'testLogSplittingAfterMasterRecoveryDueToZKExpiry,5,1365336889784.5403a557a6168e5f3c5141eb494f6088.', STARTKEY => '5', ENDKEY => '', ENCODED => 5403a557a6168e5f3c5141eb494f6088,}, server: asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:14:50,693 DEBUG [RS_OPEN_REGION-asf001.sp2.ygridcore.net,60884,1365336878389-1] handler.OpenRegionHandler(168): Opened testLogSplittingAfterMasterRecoveryDueToZKExpiry,5,1365336889784.5403a557a6168e5f3c5141eb494f6088. on server:asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:14:50,693 DEBUG [RS_OPEN_REGION-asf001.sp2.ygridcore.net,60884,1365336878389-2] zookeeper.ZKUtil(1680): regionserver:60884-0x13de46b5f8b0001 Retrieved 151 byte(s) of data from znode /hbase/region-in-transition/fb4182aef4ce07f011871ae0a083aee0; data=region=testLogSplittingAfterMasterRecoveryDueToZKExpiry,,1365336889784.fb4182aef4ce07f011871ae0a083aee0., state=RS_ZK_REGION_OPENING, servername=asf001.sp2.ygridcore.net,60884,1365336878389, createTime=1365336890590, payload.length=0 2013-04-07 12:14:50,694 INFO [PostOpenDeployTasks:55884b67b9203a71bdf8c0e3de70452d] catalog.MetaEditor(432): Updated row testLogSplittingAfterMasterRecoveryDueToZKExpiry,2,1365336889784.55884b67b9203a71bdf8c0e3de70452d. with server=asf001.sp2.ygridcore.net,43802,1365336889418 2013-04-07 12:14:50,694 INFO [PostOpenDeployTasks:55884b67b9203a71bdf8c0e3de70452d] regionserver.HRegionServer(1584): Done with post open deploy task for region=testLogSplittingAfterMasterRecoveryDueToZKExpiry,2,1365336889784.55884b67b9203a71bdf8c0e3de70452d. 2013-04-07 12:14:50,694 DEBUG [RS_OPEN_REGION-asf001.sp2.ygridcore.net,43802,1365336889418-2] zookeeper.ZKAssign(721): regionserver:43802-0x13de46b5f8b000f Attempting to transition node 55884b67b9203a71bdf8c0e3de70452d from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENED 2013-04-07 12:14:50,696 DEBUG [RS_OPEN_REGION-asf001.sp2.ygridcore.net,43802,1365336889418-2] zookeeper.ZKUtil(1680): regionserver:43802-0x13de46b5f8b000f Retrieved 152 byte(s) of data from znode /hbase/region-in-transition/55884b67b9203a71bdf8c0e3de70452d; data=region=testLogSplittingAfterMasterRecoveryDueToZKExpiry,2,1365336889784.55884b67b9203a71bdf8c0e3de70452d., state=RS_ZK_REGION_OPENING, servername=asf001.sp2.ygridcore.net,43802,1365336889418, createTime=1365336890660, payload.length=0 2013-04-07 12:14:50,697 DEBUG [RS_OPEN_REGION-asf001.sp2.ygridcore.net,43802,1365336889418-2] zookeeper.ZKAssign(784): regionserver:43802-0x13de46b5f8b000f Successfully transitioned node 55884b67b9203a71bdf8c0e3de70452d from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENED 2013-04-07 12:14:50,697 DEBUG [RS_OPEN_REGION-asf001.sp2.ygridcore.net,43802,1365336889418-2] handler.OpenRegionHandler(361): region transitioned to opened in zookeeper: {NAME => 'testLogSplittingAfterMasterRecoveryDueToZKExpiry,2,1365336889784.55884b67b9203a71bdf8c0e3de70452d.', STARTKEY => '2', ENDKEY => '3', ENCODED => 55884b67b9203a71bdf8c0e3de70452d,}, server: asf001.sp2.ygridcore.net,43802,1365336889418 2013-04-07 12:14:50,697 DEBUG [RS_OPEN_REGION-asf001.sp2.ygridcore.net,43802,1365336889418-2] handler.OpenRegionHandler(168): Opened testLogSplittingAfterMasterRecoveryDueToZKExpiry,2,1365336889784.55884b67b9203a71bdf8c0e3de70452d. on server:asf001.sp2.ygridcore.net,43802,1365336889418 2013-04-07 12:14:50,698 DEBUG [MASTER_OPEN_REGION-asf001.sp2.ygridcore.net,33796,1365336878036-0] zookeeper.ZKUtil(1680): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Retrieved 152 byte(s) of data from znode /hbase/region-in-transition/567d8ef20f90f3c1b8a889d22a1c3948; data=region=testLogSplittingAfterMasterRecoveryDueToZKExpiry,4,1365336889784.567d8ef20f90f3c1b8a889d22a1c3948., state=RS_ZK_REGION_OPENED, servername=asf001.sp2.ygridcore.net,43802,1365336889418, createTime=1365336890644, payload.length=0 2013-04-07 12:14:50,698 DEBUG [RS_OPEN_REGION-asf001.sp2.ygridcore.net,60884,1365336878389-2] zookeeper.ZKAssign(784): regionserver:60884-0x13de46b5f8b0001 Successfully transitioned node fb4182aef4ce07f011871ae0a083aee0 from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENING 2013-04-07 12:14:50,698 INFO [PostOpenDeployTasks:fb4182aef4ce07f011871ae0a083aee0] regionserver.HRegionServer(1563): Post open deploy tasks for region=testLogSplittingAfterMasterRecoveryDueToZKExpiry,,1365336889784.fb4182aef4ce07f011871ae0a083aee0. 2013-04-07 12:14:50,700 DEBUG [PostOpenDeployTasks:fb4182aef4ce07f011871ae0a083aee0] zookeeper.ZKUtil(1680): hconnection-0x43ad4a-0x13de46b5f8b0005 Retrieved 43 byte(s) of data from znode /hbase/meta-region-server; data=asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:14:50,701 DEBUG [PostOpenDeployTasks:fb4182aef4ce07f011871ae0a083aee0] zookeeper.ZKUtil(1680): hconnection-0x43ad4a-0x13de46b5f8b0005 Retrieved 43 byte(s) of data from znode /hbase/meta-region-server; data=asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:14:50,701 DEBUG [MASTER_OPEN_REGION-asf001.sp2.ygridcore.net,33796,1365336878036-1] zookeeper.ZKUtil(1680): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Retrieved 152 byte(s) of data from znode /hbase/region-in-transition/567d8ef20f90f3c1b8a889d22a1c3948; data=region=testLogSplittingAfterMasterRecoveryDueToZKExpiry,4,1365336889784.567d8ef20f90f3c1b8a889d22a1c3948., state=RS_ZK_REGION_OPENED, servername=asf001.sp2.ygridcore.net,43802,1365336889418, createTime=1365336890644, payload.length=0 2013-04-07 12:14:50,702 DEBUG [hbase-am-zkevent-worker-pool-2-thread-16] zookeeper.ZKUtil(1680): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Retrieved 152 byte(s) of data from znode /hbase/region-in-transition/43207168ac66dce85cf2a4f916007450 and set watcher; region=testLogSplittingAfterMasterRecoveryDueToZKExpiry,1,1365336889784.43207168ac66dce85cf2a4f916007450., state=RS_ZK_REGION_OPENED, servername=asf001.sp2.ygridcore.net,43802,1365336889418, createTime=1365336890654, payload.length=0 2013-04-07 12:14:50,702 DEBUG [hbase-am-zkevent-worker-pool-2-thread-16] master.AssignmentManager(740): Handling transition=RS_ZK_REGION_OPENED, server=asf001.sp2.ygridcore.net,43802,1365336889418, region=43207168ac66dce85cf2a4f916007450, current state from region state map ={testLogSplittingAfterMasterRecoveryDueToZKExpiry,1,1365336889784.43207168ac66dce85cf2a4f916007450. state=OPENING, ts=1365336890653, server=asf001.sp2.ygridcore.net,43802,1365336889418} 2013-04-07 12:14:50,823 DEBUG [Thread-214] client.MetaScanner(198): Scanning .META. starting at row=testLogSplittingAfterMasterRecoveryDueToZKExpiry,,00000000000000 for max=2147483647 rows using hconnection-0xfe04c6 2013-04-07 12:14:50,824 DEBUG [Thread-214] zookeeper.ZKUtil(1680): hconnection-0xfe04c6-0x13de46b5f8b0007 Retrieved 43 byte(s) of data from znode /hbase/meta-region-server; data=asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:14:50,829 DEBUG [Thread-214] hbase.HRegionInfo(954): Current INFO from scan results = {NAME => 'testLogSplittingAfterMasterRecoveryDueToZKExpiry,,1365336889784.fb4182aef4ce07f011871ae0a083aee0.', STARTKEY => '', ENDKEY => '1', ENCODED => fb4182aef4ce07f011871ae0a083aee0,} 2013-04-07 12:14:50,829 DEBUG [Thread-214] hbase.HRegionInfo(954): Current INFO from scan results = {NAME => 'testLogSplittingAfterMasterRecoveryDueToZKExpiry,1,1365336889784.43207168ac66dce85cf2a4f916007450.', STARTKEY => '1', ENDKEY => '2', ENCODED => 43207168ac66dce85cf2a4f916007450,} 2013-04-07 12:14:50,829 DEBUG [Thread-214] hbase.HRegionInfo(954): Current INFO from scan results = {NAME => 'testLogSplittingAfterMasterRecoveryDueToZKExpiry,2,1365336889784.55884b67b9203a71bdf8c0e3de70452d.', STARTKEY => '2', ENDKEY => '3', ENCODED => 55884b67b9203a71bdf8c0e3de70452d,} 2013-04-07 12:14:50,830 DEBUG [Thread-214] hbase.HRegionInfo(954): Current INFO from scan results = {NAME => 'testLogSplittingAfterMasterRecoveryDueToZKExpiry,3,1365336889784.4e9a3d63237d29e6bf55bf2360834cbe.', STARTKEY => '3', ENDKEY => '4', ENCODED => 4e9a3d63237d29e6bf55bf2360834cbe,} 2013-04-07 12:14:50,830 DEBUG [Thread-214] hbase.HRegionInfo(954): Current INFO from scan results = {NAME => 'testLogSplittingAfterMasterRecoveryDueToZKExpiry,4,1365336889784.567d8ef20f90f3c1b8a889d22a1c3948.', STARTKEY => '4', ENDKEY => '5', ENCODED => 567d8ef20f90f3c1b8a889d22a1c3948,} 2013-04-07 12:14:50,830 DEBUG [Thread-214] hbase.HRegionInfo(954): Current INFO from scan results = {NAME => 'testLogSplittingAfterMasterRecoveryDueToZKExpiry,5,1365336889784.5403a557a6168e5f3c5141eb494f6088.', STARTKEY => '5', ENDKEY => '', ENCODED => 5403a557a6168e5f3c5141eb494f6088,} 2013-04-07 12:14:50,833 DEBUG [Thread-214] zookeeper.ZKUtil(1680): hconnection-0xfe04c6-0x13de46b5f8b0007 Retrieved 6 byte(s) of data from znode /hbase/table/testLogSplittingAfterMasterRecoveryDueToZKExpiry; data=PBUF\x08\x00 2013-04-07 12:14:50,834 DEBUG [Thread-214] zookeeper.ZKConfig(84): Skipped reading ZK properties file 'zoo.cfg' since 'hbase.config.read.zookeeper.config' was not set to true 2013-04-07 12:14:50,834 DEBUG [Thread-214] zookeeper.ZKUtil(119): unittest opening connection to ZooKeeper with ensemble (localhost:51597) 2013-04-07 12:14:50,835 INFO [Thread-214] zookeeper.RecoverableZooKeeper(119): The identifier of this process is unittest 2013-04-07 12:14:50,839 DEBUG [Thread-214-EventThread] zookeeper.ZooKeeperWatcher(274): unittest Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2013-04-07 12:14:50,840 DEBUG [Thread-214-EventThread] zookeeper.ZooKeeperWatcher(351): unittest-0x13de46b5f8b0011 connected 2013-04-07 12:14:50,840 DEBUG [Thread-214] zookeeper.ZKUtil(1680): hconnection-0xfe04c6-0x13de46b5f8b0007 Retrieved 43 byte(s) of data from znode /hbase/meta-region-server; data=asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:14:50,841 DEBUG [Thread-214] zookeeper.ZKUtil(1680): hconnection-0xfe04c6-0x13de46b5f8b0007 Retrieved 43 byte(s) of data from znode /hbase/meta-region-server; data=asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:14:50,842 DEBUG [Thread-214] zookeeper.ZKUtil(1680): hconnection-0xfe04c6-0x13de46b5f8b0007 Retrieved 43 byte(s) of data from znode /hbase/meta-region-server; data=asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:14:50,844 DEBUG [Thread-214] client.MetaScanner(251): Current INFO from scan results = {NAME => 'testLogSplittingAfterMasterRecoveryDueToZKExpiry,,1365336889784.fb4182aef4ce07f011871ae0a083aee0.', STARTKEY => '', ENDKEY => '1', ENCODED => fb4182aef4ce07f011871ae0a083aee0,} 2013-04-07 12:14:50,844 DEBUG [Thread-214] client.MetaScanner(198): Scanning .META. starting at row=testLogSplittingAfterMasterRecoveryDueToZKExpiry,,00000000000000 for max=10 rows using hconnection-0xfe04c6 2013-04-07 12:14:50,845 DEBUG [Thread-214] zookeeper.ZKUtil(1680): hconnection-0xfe04c6-0x13de46b5f8b0007 Retrieved 43 byte(s) of data from znode /hbase/meta-region-server; data=asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:14:50,850 DEBUG [Thread-214] client.MetaScanner(251): Current INFO from scan results = {NAME => 'testLogSplittingAfterMasterRecoveryDueToZKExpiry,,1365336889784.fb4182aef4ce07f011871ae0a083aee0.', STARTKEY => '', ENDKEY => '1', ENCODED => fb4182aef4ce07f011871ae0a083aee0,} 2013-04-07 12:14:50,850 DEBUG [Thread-214] client.HConnectionManager$HConnectionImplementation(1433): Cached location for testLogSplittingAfterMasterRecoveryDueToZKExpiry,,1365336889784.fb4182aef4ce07f011871ae0a083aee0. is asf001.sp2.ygridcore.net:60884 2013-04-07 12:14:50,850 DEBUG [Thread-214] client.MetaScanner(251): Current INFO from scan results = {NAME => 'testLogSplittingAfterMasterRecoveryDueToZKExpiry,1,1365336889784.43207168ac66dce85cf2a4f916007450.', STARTKEY => '1', ENDKEY => '2', ENCODED => 43207168ac66dce85cf2a4f916007450,} 2013-04-07 12:14:50,851 DEBUG [Thread-214] client.HConnectionManager$HConnectionImplementation(1433): Cached location for testLogSplittingAfterMasterRecoveryDueToZKExpiry,1,1365336889784.43207168ac66dce85cf2a4f916007450. is asf001.sp2.ygridcore.net:43802 2013-04-07 12:14:50,851 DEBUG [Thread-214] client.MetaScanner(251): Current INFO from scan results = {NAME => 'testLogSplittingAfterMasterRecoveryDueToZKExpiry,2,1365336889784.55884b67b9203a71bdf8c0e3de70452d.', STARTKEY => '2', ENDKEY => '3', ENCODED => 55884b67b9203a71bdf8c0e3de70452d,} 2013-04-07 12:14:50,851 DEBUG [Thread-214] client.HConnectionManager$HConnectionImplementation(1433): Cached location for testLogSplittingAfterMasterRecoveryDueToZKExpiry,2,1365336889784.55884b67b9203a71bdf8c0e3de70452d. is asf001.sp2.ygridcore.net:43802 2013-04-07 12:14:50,851 DEBUG [Thread-214] client.MetaScanner(251): Current INFO from scan results = {NAME => 'testLogSplittingAfterMasterRecoveryDueToZKExpiry,3,1365336889784.4e9a3d63237d29e6bf55bf2360834cbe.', STARTKEY => '3', ENDKEY => '4', ENCODED => 4e9a3d63237d29e6bf55bf2360834cbe,} 2013-04-07 12:14:50,852 DEBUG [Thread-214] client.HConnectionManager$HConnectionImplementation(1433): Cached location for testLogSplittingAfterMasterRecoveryDueToZKExpiry,3,1365336889784.4e9a3d63237d29e6bf55bf2360834cbe. is asf001.sp2.ygridcore.net:60884 2013-04-07 12:14:50,852 DEBUG [Thread-214] client.MetaScanner(251): Current INFO from scan results = {NAME => 'testLogSplittingAfterMasterRecoveryDueToZKExpiry,4,1365336889784.567d8ef20f90f3c1b8a889d22a1c3948.', STARTKEY => '4', ENDKEY => '5', ENCODED => 567d8ef20f90f3c1b8a889d22a1c3948,} 2013-04-07 12:14:50,852 DEBUG [Thread-214] client.HConnectionManager$HConnectionImplementation(1433): Cached location for testLogSplittingAfterMasterRecoveryDueToZKExpiry,4,1365336889784.567d8ef20f90f3c1b8a889d22a1c3948. is asf001.sp2.ygridcore.net:43802 2013-04-07 12:14:50,852 DEBUG [Thread-214] client.MetaScanner(251): Current INFO from scan results = {NAME => 'testLogSplittingAfterMasterRecoveryDueToZKExpiry,5,1365336889784.5403a557a6168e5f3c5141eb494f6088.', STARTKEY => '5', ENDKEY => '', ENCODED => 5403a557a6168e5f3c5141eb494f6088,} 2013-04-07 12:14:50,853 DEBUG [Thread-214] client.HConnectionManager$HConnectionImplementation(1433): Cached location for testLogSplittingAfterMasterRecoveryDueToZKExpiry,5,1365336889784.5403a557a6168e5f3c5141eb494f6088. is asf001.sp2.ygridcore.net:60884 2013-04-07 12:14:50,856 DEBUG [IPC Reader 4 on port 60884] regionserver.QosFunction(169): Marking normal priority after getting exception=org.apache.hadoop.hbase.exceptions.NotServingRegionException: Region is not online: fb4182aef4ce07f011871ae0a083aee0 2013-04-07 12:14:50,856 DEBUG [IPC Reader 4 on port 60884] regionserver.QosFunction(169): Marking normal priority after getting exception=org.apache.hadoop.hbase.exceptions.NotServingRegionException: Region is not online: fb4182aef4ce07f011871ae0a083aee0 2013-04-07 12:14:50,858 INFO [Thread-214] client.HConnectionManager$HConnectionImplementation$ServerErrorTracker(2571): Server tracker timeout is 71373ms 2013-04-07 12:14:50,859 DEBUG [Thread-214] zookeeper.ZKUtil(1680): hconnection-0xfe04c6-0x13de46b5f8b0007 Retrieved 43 byte(s) of data from znode /hbase/meta-region-server; data=asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:14:50,860 DEBUG [Thread-214] zookeeper.ZKUtil(1680): hconnection-0xfe04c6-0x13de46b5f8b0007 Retrieved 43 byte(s) of data from znode /hbase/meta-region-server; data=asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:14:50,861 DEBUG [Thread-214] zookeeper.ZKUtil(1680): hconnection-0xfe04c6-0x13de46b5f8b0007 Retrieved 43 byte(s) of data from znode /hbase/meta-region-server; data=asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:14:50,863 DEBUG [Thread-214] client.MetaScanner(251): Current INFO from scan results = {NAME => 'testLogSplittingAfterMasterRecoveryDueToZKExpiry,,1365336889784.fb4182aef4ce07f011871ae0a083aee0.', STARTKEY => '', ENDKEY => '1', ENCODED => fb4182aef4ce07f011871ae0a083aee0,} 2013-04-07 12:14:50,863 DEBUG [Thread-214] client.MetaScanner(198): Scanning .META. starting at row=testLogSplittingAfterMasterRecoveryDueToZKExpiry,,00000000000000 for max=10 rows using hconnection-0xfe04c6 2013-04-07 12:14:50,864 DEBUG [Thread-214] zookeeper.ZKUtil(1680): hconnection-0xfe04c6-0x13de46b5f8b0007 Retrieved 43 byte(s) of data from znode /hbase/meta-region-server; data=asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:14:50,868 DEBUG [Thread-214] client.MetaScanner(251): Current INFO from scan results = {NAME => 'testLogSplittingAfterMasterRecoveryDueToZKExpiry,,1365336889784.fb4182aef4ce07f011871ae0a083aee0.', STARTKEY => '', ENDKEY => '1', ENCODED => fb4182aef4ce07f011871ae0a083aee0,} 2013-04-07 12:14:50,868 DEBUG [Thread-214] client.HConnectionManager$HConnectionImplementation(1433): Cached location for testLogSplittingAfterMasterRecoveryDueToZKExpiry,,1365336889784.fb4182aef4ce07f011871ae0a083aee0. is asf001.sp2.ygridcore.net:60884 2013-04-07 12:14:50,868 DEBUG [Thread-214] client.MetaScanner(251): Current INFO from scan results = {NAME => 'testLogSplittingAfterMasterRecoveryDueToZKExpiry,1,1365336889784.43207168ac66dce85cf2a4f916007450.', STARTKEY => '1', ENDKEY => '2', ENCODED => 43207168ac66dce85cf2a4f916007450,} 2013-04-07 12:14:50,869 DEBUG [Thread-214] client.MetaScanner(251): Current INFO from scan results = {NAME => 'testLogSplittingAfterMasterRecoveryDueToZKExpiry,2,1365336889784.55884b67b9203a71bdf8c0e3de70452d.', STARTKEY => '2', ENDKEY => '3', ENCODED => 55884b67b9203a71bdf8c0e3de70452d,} 2013-04-07 12:14:50,869 DEBUG [Thread-214] client.MetaScanner(251): Current INFO from scan results = {NAME => 'testLogSplittingAfterMasterRecoveryDueToZKExpiry,3,1365336889784.4e9a3d63237d29e6bf55bf2360834cbe.', STARTKEY => '3', ENDKEY => '4', ENCODED => 4e9a3d63237d29e6bf55bf2360834cbe,} 2013-04-07 12:14:50,870 DEBUG [Thread-214] client.MetaScanner(251): Current INFO from scan results = {NAME => 'testLogSplittingAfterMasterRecoveryDueToZKExpiry,4,1365336889784.567d8ef20f90f3c1b8a889d22a1c3948.', STARTKEY => '4', ENDKEY => '5', ENCODED => 567d8ef20f90f3c1b8a889d22a1c3948,} 2013-04-07 12:14:50,870 DEBUG [Thread-214] client.MetaScanner(251): Current INFO from scan results = {NAME => 'testLogSplittingAfterMasterRecoveryDueToZKExpiry,5,1365336889784.5403a557a6168e5f3c5141eb494f6088.', STARTKEY => '5', ENDKEY => '', ENCODED => 5403a557a6168e5f3c5141eb494f6088,} 2013-04-07 12:14:51,859 DEBUG [IPC Reader 4 on port 60884] regionserver.QosFunction(169): Marking normal priority after getting exception=org.apache.hadoop.hbase.exceptions.NotServingRegionException: Region is not online: fb4182aef4ce07f011871ae0a083aee0 2013-04-07 12:14:51,859 DEBUG [IPC Reader 4 on port 60884] regionserver.QosFunction(169): Marking normal priority after getting exception=org.apache.hadoop.hbase.exceptions.NotServingRegionException: Region is not online: fb4182aef4ce07f011871ae0a083aee0 2013-04-07 12:14:51,862 DEBUG [Thread-214] zookeeper.ZKUtil(1680): hconnection-0xfe04c6-0x13de46b5f8b0007 Retrieved 43 byte(s) of data from znode /hbase/meta-region-server; data=asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:14:51,863 DEBUG [Thread-214] zookeeper.ZKUtil(1680): hconnection-0xfe04c6-0x13de46b5f8b0007 Retrieved 43 byte(s) of data from znode /hbase/meta-region-server; data=asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:14:51,864 DEBUG [Thread-214] zookeeper.ZKUtil(1680): hconnection-0xfe04c6-0x13de46b5f8b0007 Retrieved 43 byte(s) of data from znode /hbase/meta-region-server; data=asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:14:51,867 DEBUG [Thread-214] client.MetaScanner(251): Current INFO from scan results = {NAME => 'testLogSplittingAfterMasterRecoveryDueToZKExpiry,,1365336889784.fb4182aef4ce07f011871ae0a083aee0.', STARTKEY => '', ENDKEY => '1', ENCODED => fb4182aef4ce07f011871ae0a083aee0,} 2013-04-07 12:14:51,867 DEBUG [Thread-214] client.MetaScanner(198): Scanning .META. starting at row=testLogSplittingAfterMasterRecoveryDueToZKExpiry,,00000000000000 for max=10 rows using hconnection-0xfe04c6 2013-04-07 12:14:51,868 DEBUG [Thread-214] zookeeper.ZKUtil(1680): hconnection-0xfe04c6-0x13de46b5f8b0007 Retrieved 43 byte(s) of data from znode /hbase/meta-region-server; data=asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:14:51,872 DEBUG [Thread-214] client.MetaScanner(251): Current INFO from scan results = {NAME => 'testLogSplittingAfterMasterRecoveryDueToZKExpiry,,1365336889784.fb4182aef4ce07f011871ae0a083aee0.', STARTKEY => '', ENDKEY => '1', ENCODED => fb4182aef4ce07f011871ae0a083aee0,} 2013-04-07 12:14:51,873 DEBUG [Thread-214] client.HConnectionManager$HConnectionImplementation(1433): Cached location for testLogSplittingAfterMasterRecoveryDueToZKExpiry,,1365336889784.fb4182aef4ce07f011871ae0a083aee0. is asf001.sp2.ygridcore.net:60884 2013-04-07 12:14:51,873 DEBUG [Thread-214] client.MetaScanner(251): Current INFO from scan results = {NAME => 'testLogSplittingAfterMasterRecoveryDueToZKExpiry,1,1365336889784.43207168ac66dce85cf2a4f916007450.', STARTKEY => '1', ENDKEY => '2', ENCODED => 43207168ac66dce85cf2a4f916007450,} 2013-04-07 12:14:51,875 DEBUG [Thread-214] client.MetaScanner(251): Current INFO from scan results = {NAME => 'testLogSplittingAfterMasterRecoveryDueToZKExpiry,2,1365336889784.55884b67b9203a71bdf8c0e3de70452d.', STARTKEY => '2', ENDKEY => '3', ENCODED => 55884b67b9203a71bdf8c0e3de70452d,} 2013-04-07 12:14:51,876 DEBUG [Thread-214] client.MetaScanner(251): Current INFO from scan results = {NAME => 'testLogSplittingAfterMasterRecoveryDueToZKExpiry,3,1365336889784.4e9a3d63237d29e6bf55bf2360834cbe.', STARTKEY => '3', ENDKEY => '4', ENCODED => 4e9a3d63237d29e6bf55bf2360834cbe,} 2013-04-07 12:14:51,876 DEBUG [Thread-214] client.MetaScanner(251): Current INFO from scan results = {NAME => 'testLogSplittingAfterMasterRecoveryDueToZKExpiry,4,1365336889784.567d8ef20f90f3c1b8a889d22a1c3948.', STARTKEY => '4', ENDKEY => '5', ENCODED => 567d8ef20f90f3c1b8a889d22a1c3948,} 2013-04-07 12:14:51,876 DEBUG [Thread-214] client.MetaScanner(251): Current INFO from scan results = {NAME => 'testLogSplittingAfterMasterRecoveryDueToZKExpiry,5,1365336889784.5403a557a6168e5f3c5141eb494f6088.', STARTKEY => '5', ENDKEY => '', ENCODED => 5403a557a6168e5f3c5141eb494f6088,} 2013-04-07 12:14:52,869 DEBUG [IPC Reader 4 on port 60884] regionserver.QosFunction(169): Marking normal priority after getting exception=org.apache.hadoop.hbase.exceptions.NotServingRegionException: Region is not online: fb4182aef4ce07f011871ae0a083aee0 2013-04-07 12:14:52,869 DEBUG [IPC Reader 4 on port 60884] regionserver.QosFunction(169): Marking normal priority after getting exception=org.apache.hadoop.hbase.exceptions.NotServingRegionException: Region is not online: fb4182aef4ce07f011871ae0a083aee0 2013-04-07 12:14:52,873 DEBUG [Thread-214] zookeeper.ZKUtil(1680): hconnection-0xfe04c6-0x13de46b5f8b0007 Retrieved 43 byte(s) of data from znode /hbase/meta-region-server; data=asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:14:52,874 DEBUG [Thread-214] zookeeper.ZKUtil(1680): hconnection-0xfe04c6-0x13de46b5f8b0007 Retrieved 43 byte(s) of data from znode /hbase/meta-region-server; data=asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:14:52,875 DEBUG [Thread-214] zookeeper.ZKUtil(1680): hconnection-0xfe04c6-0x13de46b5f8b0007 Retrieved 43 byte(s) of data from znode /hbase/meta-region-server; data=asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:14:52,877 DEBUG [Thread-214] client.MetaScanner(251): Current INFO from scan results = {NAME => 'testLogSplittingAfterMasterRecoveryDueToZKExpiry,,1365336889784.fb4182aef4ce07f011871ae0a083aee0.', STARTKEY => '', ENDKEY => '1', ENCODED => fb4182aef4ce07f011871ae0a083aee0,} 2013-04-07 12:14:52,877 DEBUG [Thread-214] client.MetaScanner(198): Scanning .META. starting at row=testLogSplittingAfterMasterRecoveryDueToZKExpiry,,00000000000000 for max=10 rows using hconnection-0xfe04c6 2013-04-07 12:14:52,878 DEBUG [Thread-214] zookeeper.ZKUtil(1680): hconnection-0xfe04c6-0x13de46b5f8b0007 Retrieved 43 byte(s) of data from znode /hbase/meta-region-server; data=asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:14:52,886 DEBUG [Thread-214] client.MetaScanner(251): Current INFO from scan results = {NAME => 'testLogSplittingAfterMasterRecoveryDueToZKExpiry,,1365336889784.fb4182aef4ce07f011871ae0a083aee0.', STARTKEY => '', ENDKEY => '1', ENCODED => fb4182aef4ce07f011871ae0a083aee0,} 2013-04-07 12:14:52,886 DEBUG [Thread-214] client.HConnectionManager$HConnectionImplementation(1433): Cached location for testLogSplittingAfterMasterRecoveryDueToZKExpiry,,1365336889784.fb4182aef4ce07f011871ae0a083aee0. is asf001.sp2.ygridcore.net:60884 2013-04-07 12:14:52,887 DEBUG [Thread-214] client.MetaScanner(251): Current INFO from scan results = {NAME => 'testLogSplittingAfterMasterRecoveryDueToZKExpiry,1,1365336889784.43207168ac66dce85cf2a4f916007450.', STARTKEY => '1', ENDKEY => '2', ENCODED => 43207168ac66dce85cf2a4f916007450,} 2013-04-07 12:14:52,887 DEBUG [Thread-214] client.MetaScanner(251): Current INFO from scan results = {NAME => 'testLogSplittingAfterMasterRecoveryDueToZKExpiry,2,1365336889784.55884b67b9203a71bdf8c0e3de70452d.', STARTKEY => '2', ENDKEY => '3', ENCODED => 55884b67b9203a71bdf8c0e3de70452d,} 2013-04-07 12:14:52,887 DEBUG [Thread-214] client.MetaScanner(251): Current INFO from scan results = {NAME => 'testLogSplittingAfterMasterRecoveryDueToZKExpiry,3,1365336889784.4e9a3d63237d29e6bf55bf2360834cbe.', STARTKEY => '3', ENDKEY => '4', ENCODED => 4e9a3d63237d29e6bf55bf2360834cbe,} 2013-04-07 12:14:52,888 DEBUG [Thread-214] client.MetaScanner(251): Current INFO from scan results = {NAME => 'testLogSplittingAfterMasterRecoveryDueToZKExpiry,4,1365336889784.567d8ef20f90f3c1b8a889d22a1c3948.', STARTKEY => '4', ENDKEY => '5', ENCODED => 567d8ef20f90f3c1b8a889d22a1c3948,} 2013-04-07 12:14:52,888 DEBUG [Thread-214] client.MetaScanner(251): Current INFO from scan results = {NAME => 'testLogSplittingAfterMasterRecoveryDueToZKExpiry,5,1365336889784.5403a557a6168e5f3c5141eb494f6088.', STARTKEY => '5', ENDKEY => '', ENCODED => 5403a557a6168e5f3c5141eb494f6088,} 2013-04-07 12:14:53,874 DEBUG [IPC Reader 4 on port 60884] regionserver.QosFunction(169): Marking normal priority after getting exception=org.apache.hadoop.hbase.exceptions.NotServingRegionException: Region is not online: fb4182aef4ce07f011871ae0a083aee0 2013-04-07 12:14:53,874 DEBUG [IPC Reader 4 on port 60884] regionserver.QosFunction(169): Marking normal priority after getting exception=org.apache.hadoop.hbase.exceptions.NotServingRegionException: Region is not online: fb4182aef4ce07f011871ae0a083aee0 2013-04-07 12:14:53,878 DEBUG [Thread-214] zookeeper.ZKUtil(1680): hconnection-0xfe04c6-0x13de46b5f8b0007 Retrieved 43 byte(s) of data from znode /hbase/meta-region-server; data=asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:14:53,879 DEBUG [Thread-214] zookeeper.ZKUtil(1680): hconnection-0xfe04c6-0x13de46b5f8b0007 Retrieved 43 byte(s) of data from znode /hbase/meta-region-server; data=asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:14:53,880 DEBUG [Thread-214] zookeeper.ZKUtil(1680): hconnection-0xfe04c6-0x13de46b5f8b0007 Retrieved 43 byte(s) of data from znode /hbase/meta-region-server; data=asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:14:53,884 DEBUG [Thread-214] client.MetaScanner(251): Current INFO from scan results = {NAME => 'testLogSplittingAfterMasterRecoveryDueToZKExpiry,,1365336889784.fb4182aef4ce07f011871ae0a083aee0.', STARTKEY => '', ENDKEY => '1', ENCODED => fb4182aef4ce07f011871ae0a083aee0,} 2013-04-07 12:14:53,884 DEBUG [Thread-214] client.MetaScanner(198): Scanning .META. starting at row=testLogSplittingAfterMasterRecoveryDueToZKExpiry,,00000000000000 for max=10 rows using hconnection-0xfe04c6 2013-04-07 12:14:53,885 DEBUG [Thread-214] zookeeper.ZKUtil(1680): hconnection-0xfe04c6-0x13de46b5f8b0007 Retrieved 43 byte(s) of data from znode /hbase/meta-region-server; data=asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:14:53,890 DEBUG [Thread-214] client.MetaScanner(251): Current INFO from scan results = {NAME => 'testLogSplittingAfterMasterRecoveryDueToZKExpiry,,1365336889784.fb4182aef4ce07f011871ae0a083aee0.', STARTKEY => '', ENDKEY => '1', ENCODED => fb4182aef4ce07f011871ae0a083aee0,} 2013-04-07 12:14:53,890 DEBUG [Thread-214] client.HConnectionManager$HConnectionImplementation(1433): Cached location for testLogSplittingAfterMasterRecoveryDueToZKExpiry,,1365336889784.fb4182aef4ce07f011871ae0a083aee0. is asf001.sp2.ygridcore.net:60884 2013-04-07 12:14:53,890 DEBUG [Thread-214] client.MetaScanner(251): Current INFO from scan results = {NAME => 'testLogSplittingAfterMasterRecoveryDueToZKExpiry,1,1365336889784.43207168ac66dce85cf2a4f916007450.', STARTKEY => '1', ENDKEY => '2', ENCODED => 43207168ac66dce85cf2a4f916007450,} 2013-04-07 12:14:53,891 DEBUG [Thread-214] client.MetaScanner(251): Current INFO from scan results = {NAME => 'testLogSplittingAfterMasterRecoveryDueToZKExpiry,2,1365336889784.55884b67b9203a71bdf8c0e3de70452d.', STARTKEY => '2', ENDKEY => '3', ENCODED => 55884b67b9203a71bdf8c0e3de70452d,} 2013-04-07 12:14:53,891 DEBUG [Thread-214] client.MetaScanner(251): Current INFO from scan results = {NAME => 'testLogSplittingAfterMasterRecoveryDueToZKExpiry,3,1365336889784.4e9a3d63237d29e6bf55bf2360834cbe.', STARTKEY => '3', ENDKEY => '4', ENCODED => 4e9a3d63237d29e6bf55bf2360834cbe,} 2013-04-07 12:14:53,891 DEBUG [Thread-214] client.MetaScanner(251): Current INFO from scan results = {NAME => 'testLogSplittingAfterMasterRecoveryDueToZKExpiry,4,1365336889784.567d8ef20f90f3c1b8a889d22a1c3948.', STARTKEY => '4', ENDKEY => '5', ENCODED => 567d8ef20f90f3c1b8a889d22a1c3948,} 2013-04-07 12:14:53,892 DEBUG [Thread-214] client.MetaScanner(251): Current INFO from scan results = {NAME => 'testLogSplittingAfterMasterRecoveryDueToZKExpiry,5,1365336889784.5403a557a6168e5f3c5141eb494f6088.', STARTKEY => '5', ENDKEY => '', ENCODED => 5403a557a6168e5f3c5141eb494f6088,} 2013-04-07 12:14:55,885 DEBUG [IPC Reader 4 on port 60884] regionserver.QosFunction(169): Marking normal priority after getting exception=org.apache.hadoop.hbase.exceptions.NotServingRegionException: Region is not online: fb4182aef4ce07f011871ae0a083aee0 2013-04-07 12:14:55,886 DEBUG [IPC Reader 4 on port 60884] regionserver.QosFunction(169): Marking normal priority after getting exception=org.apache.hadoop.hbase.exceptions.NotServingRegionException: Region is not online: fb4182aef4ce07f011871ae0a083aee0 2013-04-07 12:14:55,888 DEBUG [Thread-214] zookeeper.ZKUtil(1680): hconnection-0xfe04c6-0x13de46b5f8b0007 Retrieved 43 byte(s) of data from znode /hbase/meta-region-server; data=asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:14:55,890 DEBUG [Thread-214] zookeeper.ZKUtil(1680): hconnection-0xfe04c6-0x13de46b5f8b0007 Retrieved 43 byte(s) of data from znode /hbase/meta-region-server; data=asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:14:55,891 DEBUG [Thread-214] zookeeper.ZKUtil(1680): hconnection-0xfe04c6-0x13de46b5f8b0007 Retrieved 43 byte(s) of data from znode /hbase/meta-region-server; data=asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:14:55,893 DEBUG [Thread-214] client.MetaScanner(251): Current INFO from scan results = {NAME => 'testLogSplittingAfterMasterRecoveryDueToZKExpiry,,1365336889784.fb4182aef4ce07f011871ae0a083aee0.', STARTKEY => '', ENDKEY => '1', ENCODED => fb4182aef4ce07f011871ae0a083aee0,} 2013-04-07 12:14:55,893 DEBUG [Thread-214] client.MetaScanner(198): Scanning .META. starting at row=testLogSplittingAfterMasterRecoveryDueToZKExpiry,,00000000000000 for max=10 rows using hconnection-0xfe04c6 2013-04-07 12:14:55,894 DEBUG [Thread-214] zookeeper.ZKUtil(1680): hconnection-0xfe04c6-0x13de46b5f8b0007 Retrieved 43 byte(s) of data from znode /hbase/meta-region-server; data=asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:14:55,898 DEBUG [Thread-214] client.MetaScanner(251): Current INFO from scan results = {NAME => 'testLogSplittingAfterMasterRecoveryDueToZKExpiry,,1365336889784.fb4182aef4ce07f011871ae0a083aee0.', STARTKEY => '', ENDKEY => '1', ENCODED => fb4182aef4ce07f011871ae0a083aee0,} 2013-04-07 12:14:55,899 DEBUG [Thread-214] client.HConnectionManager$HConnectionImplementation(1433): Cached location for testLogSplittingAfterMasterRecoveryDueToZKExpiry,,1365336889784.fb4182aef4ce07f011871ae0a083aee0. is asf001.sp2.ygridcore.net:60884 2013-04-07 12:14:55,899 DEBUG [Thread-214] client.MetaScanner(251): Current INFO from scan results = {NAME => 'testLogSplittingAfterMasterRecoveryDueToZKExpiry,1,1365336889784.43207168ac66dce85cf2a4f916007450.', STARTKEY => '1', ENDKEY => '2', ENCODED => 43207168ac66dce85cf2a4f916007450,} 2013-04-07 12:14:55,899 DEBUG [Thread-214] client.MetaScanner(251): Current INFO from scan results = {NAME => 'testLogSplittingAfterMasterRecoveryDueToZKExpiry,2,1365336889784.55884b67b9203a71bdf8c0e3de70452d.', STARTKEY => '2', ENDKEY => '3', ENCODED => 55884b67b9203a71bdf8c0e3de70452d,} 2013-04-07 12:14:55,900 DEBUG [Thread-214] client.MetaScanner(251): Current INFO from scan results = {NAME => 'testLogSplittingAfterMasterRecoveryDueToZKExpiry,3,1365336889784.4e9a3d63237d29e6bf55bf2360834cbe.', STARTKEY => '3', ENDKEY => '4', ENCODED => 4e9a3d63237d29e6bf55bf2360834cbe,} 2013-04-07 12:14:55,900 DEBUG [Thread-214] client.MetaScanner(251): Current INFO from scan results = {NAME => 'testLogSplittingAfterMasterRecoveryDueToZKExpiry,4,1365336889784.567d8ef20f90f3c1b8a889d22a1c3948.', STARTKEY => '4', ENDKEY => '5', ENCODED => 567d8ef20f90f3c1b8a889d22a1c3948,} 2013-04-07 12:14:55,900 DEBUG [Thread-214] client.MetaScanner(251): Current INFO from scan results = {NAME => 'testLogSplittingAfterMasterRecoveryDueToZKExpiry,5,1365336889784.5403a557a6168e5f3c5141eb494f6088.', STARTKEY => '5', ENDKEY => '', ENCODED => 5403a557a6168e5f3c5141eb494f6088,} 2013-04-07 12:14:56,426 WARN [RegionServer:1;asf001.sp2.ygridcore.net,35684,1365336878445] zookeeper.RecoverableZooKeeper(236): Possibly transient ZooKeeper, quorum=localhost:51597, exception=org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /hbase/rs/asf001.sp2.ygridcore.net,35684,1365336878445 2013-04-07 12:14:56,426 ERROR [RegionServer:1;asf001.sp2.ygridcore.net,35684,1365336878445] zookeeper.RecoverableZooKeeper(238): ZooKeeper delete failed after 3 retries 2013-04-07 12:14:56,426 WARN [RegionServer:1;asf001.sp2.ygridcore.net,35684,1365336878445] regionserver.HRegionServer(882): Failed deleting my ephemeral node org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired for /hbase/rs/asf001.sp2.ygridcore.net,35684,1365336878445 at org.apache.zookeeper.KeeperException.create(KeeperException.java:127) at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) at org.apache.zookeeper.ZooKeeper.delete(ZooKeeper.java:873) at org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.delete(RecoverableZooKeeper.java:151) at org.apache.hadoop.hbase.zookeeper.ZKUtil.deleteNode(ZKUtil.java:1259) at org.apache.hadoop.hbase.zookeeper.ZKUtil.deleteNode(ZKUtil.java:1248) at org.apache.hadoop.hbase.regionserver.HRegionServer.deleteMyEphemeralNode(HRegionServer.java:1136) at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:880) at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.runRegionServer(MiniHBaseCluster.java:151) at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.access$000(MiniHBaseCluster.java:103) at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer$1.run(MiniHBaseCluster.java:135) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:337) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1131) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.hbase.util.Methods.call(Methods.java:41) at org.apache.hadoop.hbase.security.User.call(User.java:420) at org.apache.hadoop.hbase.security.User.access$300(User.java:51) at org.apache.hadoop.hbase.security.User$SecureHadoopUser.runAs(User.java:260) at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.run(MiniHBaseCluster.java:133) at java.lang.Thread.run(Thread.java:662) 2013-04-07 12:14:56,427 INFO [RegionServer:1;asf001.sp2.ygridcore.net,35684,1365336878445] regionserver.HRegionServer(888): stopping server asf001.sp2.ygridcore.net,35684,1365336878445; zookeeper connection closed. 2013-04-07 12:14:56,427 INFO [RegionServer:1;asf001.sp2.ygridcore.net,35684,1365336878445] regionserver.HRegionServer(891): RegionServer:1;asf001.sp2.ygridcore.net,35684,1365336878445 exiting 2013-04-07 12:14:56,427 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@15a4a77] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(186): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@15a4a77 2013-04-07 12:14:56,427 ERROR [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@15a4a77] hdfs.DFSClient(416): Failed to close file /user/jenkins/hbase/.logs/asf001.sp2.ygridcore.net,35684,1365336878445/asf001.sp2.ygridcore.net%2C35684%2C1365336878445.1365336879032 org.apache.hadoop.ipc.RemoteException: org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException: No lease on /user/jenkins/hbase/.logs/asf001.sp2.ygridcore.net,35684,1365336878445/asf001.sp2.ygridcore.net%2C35684%2C1365336878445.1365336879032 File does not exist. [Lease. Holder: DFSClient_hb_rs_asf001.sp2.ygridcore.net,35684,1365336878445_-2003770590_156, pendingcreates: 1] at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:1720) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:1711) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1619) at org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:736) at sun.reflect.GeneratedMethodAccessor24.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:578) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1393) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1389) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1149) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1387) at org.apache.hadoop.ipc.Client.call(Client.java:1107) at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:229) at $Proxy9.addBlock(Unknown Source) at sun.reflect.GeneratedMethodAccessor24.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:85) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:62) at $Proxy9.addBlock(Unknown Source) at sun.reflect.GeneratedMethodAccessor24.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:267) at $Proxy18.addBlock(Unknown Source) at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:3686) at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:3546) at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2600(DFSClient.java:2749) at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2989) 2013-04-07 12:14:56,428 ERROR [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@15a4a77] hdfs.DFSClient(416): Failed to close file /user/jenkins/hbase/.logs/asf001.sp2.ygridcore.net,35684,1365336878445/asf001.sp2.ygridcore.net%2C35684%2C1365336878445.1365336880146.meta org.apache.hadoop.ipc.RemoteException: org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException: No lease on /user/jenkins/hbase/.logs/asf001.sp2.ygridcore.net,35684,1365336878445/asf001.sp2.ygridcore.net%2C35684%2C1365336878445.1365336880146.meta File does not exist. [Lease. Holder: DFSClient_hb_rs_asf001.sp2.ygridcore.net,35684,1365336878445_-2003770590_156, pendingcreates: 1] at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:1720) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:1711) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1619) at org.apache.hadoop.hdfs.server.namenode.NameNode.addBlock(NameNode.java:736) at sun.reflect.GeneratedMethodAccessor24.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:578) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1393) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1389) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1149) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1387) at org.apache.hadoop.ipc.Client.call(Client.java:1107) at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:229) at $Proxy9.addBlock(Unknown Source) at sun.reflect.GeneratedMethodAccessor24.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:85) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:62) at $Proxy9.addBlock(Unknown Source) at sun.reflect.GeneratedMethodAccessor24.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:267) at $Proxy18.addBlock(Unknown Source) at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:3686) at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:3546) at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2600(DFSClient.java:2749) at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2989) 2013-04-07 12:14:57,894 DEBUG [IPC Reader 4 on port 60884] regionserver.QosFunction(169): Marking normal priority after getting exception=org.apache.hadoop.hbase.exceptions.NotServingRegionException: Region is not online: fb4182aef4ce07f011871ae0a083aee0 2013-04-07 12:14:57,894 DEBUG [IPC Reader 4 on port 60884] regionserver.QosFunction(169): Marking normal priority after getting exception=org.apache.hadoop.hbase.exceptions.NotServingRegionException: Region is not online: fb4182aef4ce07f011871ae0a083aee0 2013-04-07 12:14:57,897 DEBUG [Thread-214] zookeeper.ZKUtil(1680): hconnection-0xfe04c6-0x13de46b5f8b0007 Retrieved 43 byte(s) of data from znode /hbase/meta-region-server; data=asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:14:57,899 DEBUG [Thread-214] zookeeper.ZKUtil(1680): hconnection-0xfe04c6-0x13de46b5f8b0007 Retrieved 43 byte(s) of data from znode /hbase/meta-region-server; data=asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:14:57,900 DEBUG [Thread-214] zookeeper.ZKUtil(1680): hconnection-0xfe04c6-0x13de46b5f8b0007 Retrieved 43 byte(s) of data from znode /hbase/meta-region-server; data=asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:14:57,902 DEBUG [Thread-214] client.MetaScanner(251): Current INFO from scan results = {NAME => 'testLogSplittingAfterMasterRecoveryDueToZKExpiry,,1365336889784.fb4182aef4ce07f011871ae0a083aee0.', STARTKEY => '', ENDKEY => '1', ENCODED => fb4182aef4ce07f011871ae0a083aee0,} 2013-04-07 12:14:57,902 DEBUG [Thread-214] client.MetaScanner(198): Scanning .META. starting at row=testLogSplittingAfterMasterRecoveryDueToZKExpiry,,00000000000000 for max=10 rows using hconnection-0xfe04c6 2013-04-07 12:14:57,903 DEBUG [Thread-214] zookeeper.ZKUtil(1680): hconnection-0xfe04c6-0x13de46b5f8b0007 Retrieved 43 byte(s) of data from znode /hbase/meta-region-server; data=asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:14:57,907 DEBUG [Thread-214] client.MetaScanner(251): Current INFO from scan results = {NAME => 'testLogSplittingAfterMasterRecoveryDueToZKExpiry,,1365336889784.fb4182aef4ce07f011871ae0a083aee0.', STARTKEY => '', ENDKEY => '1', ENCODED => fb4182aef4ce07f011871ae0a083aee0,} 2013-04-07 12:14:57,908 DEBUG [Thread-214] client.HConnectionManager$HConnectionImplementation(1433): Cached location for testLogSplittingAfterMasterRecoveryDueToZKExpiry,,1365336889784.fb4182aef4ce07f011871ae0a083aee0. is asf001.sp2.ygridcore.net:60884 2013-04-07 12:14:57,908 DEBUG [Thread-214] client.MetaScanner(251): Current INFO from scan results = {NAME => 'testLogSplittingAfterMasterRecoveryDueToZKExpiry,1,1365336889784.43207168ac66dce85cf2a4f916007450.', STARTKEY => '1', ENDKEY => '2', ENCODED => 43207168ac66dce85cf2a4f916007450,} 2013-04-07 12:14:57,908 DEBUG [Thread-214] client.MetaScanner(251): Current INFO from scan results = {NAME => 'testLogSplittingAfterMasterRecoveryDueToZKExpiry,2,1365336889784.55884b67b9203a71bdf8c0e3de70452d.', STARTKEY => '2', ENDKEY => '3', ENCODED => 55884b67b9203a71bdf8c0e3de70452d,} 2013-04-07 12:14:57,909 DEBUG [Thread-214] client.MetaScanner(251): Current INFO from scan results = {NAME => 'testLogSplittingAfterMasterRecoveryDueToZKExpiry,3,1365336889784.4e9a3d63237d29e6bf55bf2360834cbe.', STARTKEY => '3', ENDKEY => '4', ENCODED => 4e9a3d63237d29e6bf55bf2360834cbe,} 2013-04-07 12:14:57,909 DEBUG [Thread-214] client.MetaScanner(251): Current INFO from scan results = {NAME => 'testLogSplittingAfterMasterRecoveryDueToZKExpiry,4,1365336889784.567d8ef20f90f3c1b8a889d22a1c3948.', STARTKEY => '4', ENDKEY => '5', ENCODED => 567d8ef20f90f3c1b8a889d22a1c3948,} 2013-04-07 12:14:57,909 DEBUG [Thread-214] client.MetaScanner(251): Current INFO from scan results = {NAME => 'testLogSplittingAfterMasterRecoveryDueToZKExpiry,5,1365336889784.5403a557a6168e5f3c5141eb494f6088.', STARTKEY => '5', ENDKEY => '', ENCODED => 5403a557a6168e5f3c5141eb494f6088,} 2013-04-07 12:15:01,919 DEBUG [IPC Reader 4 on port 60884] regionserver.QosFunction(169): Marking normal priority after getting exception=org.apache.hadoop.hbase.exceptions.NotServingRegionException: Region is not online: fb4182aef4ce07f011871ae0a083aee0 2013-04-07 12:15:01,919 DEBUG [IPC Reader 4 on port 60884] regionserver.QosFunction(169): Marking normal priority after getting exception=org.apache.hadoop.hbase.exceptions.NotServingRegionException: Region is not online: fb4182aef4ce07f011871ae0a083aee0 2013-04-07 12:15:01,922 DEBUG [Thread-214] zookeeper.ZKUtil(1680): hconnection-0xfe04c6-0x13de46b5f8b0007 Retrieved 43 byte(s) of data from znode /hbase/meta-region-server; data=asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:15:01,923 DEBUG [Thread-214] zookeeper.ZKUtil(1680): hconnection-0xfe04c6-0x13de46b5f8b0007 Retrieved 43 byte(s) of data from znode /hbase/meta-region-server; data=asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:15:01,924 DEBUG [Thread-214] zookeeper.ZKUtil(1680): hconnection-0xfe04c6-0x13de46b5f8b0007 Retrieved 43 byte(s) of data from znode /hbase/meta-region-server; data=asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:15:01,925 DEBUG [Thread-214] client.MetaScanner(251): Current INFO from scan results = {NAME => 'testLogSplittingAfterMasterRecoveryDueToZKExpiry,,1365336889784.fb4182aef4ce07f011871ae0a083aee0.', STARTKEY => '', ENDKEY => '1', ENCODED => fb4182aef4ce07f011871ae0a083aee0,} 2013-04-07 12:15:01,926 DEBUG [Thread-214] client.MetaScanner(198): Scanning .META. starting at row=testLogSplittingAfterMasterRecoveryDueToZKExpiry,,00000000000000 for max=10 rows using hconnection-0xfe04c6 2013-04-07 12:15:01,926 DEBUG [Thread-214] zookeeper.ZKUtil(1680): hconnection-0xfe04c6-0x13de46b5f8b0007 Retrieved 43 byte(s) of data from znode /hbase/meta-region-server; data=asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:15:01,930 DEBUG [Thread-214] client.MetaScanner(251): Current INFO from scan results = {NAME => 'testLogSplittingAfterMasterRecoveryDueToZKExpiry,,1365336889784.fb4182aef4ce07f011871ae0a083aee0.', STARTKEY => '', ENDKEY => '1', ENCODED => fb4182aef4ce07f011871ae0a083aee0,} 2013-04-07 12:15:01,931 DEBUG [Thread-214] client.HConnectionManager$HConnectionImplementation(1433): Cached location for testLogSplittingAfterMasterRecoveryDueToZKExpiry,,1365336889784.fb4182aef4ce07f011871ae0a083aee0. is asf001.sp2.ygridcore.net:60884 2013-04-07 12:15:01,931 DEBUG [Thread-214] client.MetaScanner(251): Current INFO from scan results = {NAME => 'testLogSplittingAfterMasterRecoveryDueToZKExpiry,1,1365336889784.43207168ac66dce85cf2a4f916007450.', STARTKEY => '1', ENDKEY => '2', ENCODED => 43207168ac66dce85cf2a4f916007450,} 2013-04-07 12:15:01,931 DEBUG [Thread-214] client.MetaScanner(251): Current INFO from scan results = {NAME => 'testLogSplittingAfterMasterRecoveryDueToZKExpiry,2,1365336889784.55884b67b9203a71bdf8c0e3de70452d.', STARTKEY => '2', ENDKEY => '3', ENCODED => 55884b67b9203a71bdf8c0e3de70452d,} 2013-04-07 12:15:01,932 DEBUG [Thread-214] client.MetaScanner(251): Current INFO from scan results = {NAME => 'testLogSplittingAfterMasterRecoveryDueToZKExpiry,3,1365336889784.4e9a3d63237d29e6bf55bf2360834cbe.', STARTKEY => '3', ENDKEY => '4', ENCODED => 4e9a3d63237d29e6bf55bf2360834cbe,} 2013-04-07 12:15:01,932 DEBUG [Thread-214] client.MetaScanner(251): Current INFO from scan results = {NAME => 'testLogSplittingAfterMasterRecoveryDueToZKExpiry,4,1365336889784.567d8ef20f90f3c1b8a889d22a1c3948.', STARTKEY => '4', ENDKEY => '5', ENCODED => 567d8ef20f90f3c1b8a889d22a1c3948,} 2013-04-07 12:15:01,932 DEBUG [Thread-214] client.MetaScanner(251): Current INFO from scan results = {NAME => 'testLogSplittingAfterMasterRecoveryDueToZKExpiry,5,1365336889784.5403a557a6168e5f3c5141eb494f6088.', STARTKEY => '5', ENDKEY => '', ENCODED => 5403a557a6168e5f3c5141eb494f6088,} 2013-04-07 12:15:05,930 DEBUG [IPC Reader 4 on port 60884] regionserver.QosFunction(169): Marking normal priority after getting exception=org.apache.hadoop.hbase.exceptions.NotServingRegionException: Region is not online: fb4182aef4ce07f011871ae0a083aee0 2013-04-07 12:15:05,930 DEBUG [IPC Reader 4 on port 60884] regionserver.QosFunction(169): Marking normal priority after getting exception=org.apache.hadoop.hbase.exceptions.NotServingRegionException: Region is not online: fb4182aef4ce07f011871ae0a083aee0 2013-04-07 12:15:05,933 DEBUG [Thread-214] zookeeper.ZKUtil(1680): hconnection-0xfe04c6-0x13de46b5f8b0007 Retrieved 43 byte(s) of data from znode /hbase/meta-region-server; data=asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:15:05,934 DEBUG [Thread-214] zookeeper.ZKUtil(1680): hconnection-0xfe04c6-0x13de46b5f8b0007 Retrieved 43 byte(s) of data from znode /hbase/meta-region-server; data=asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:15:05,935 DEBUG [Thread-214] zookeeper.ZKUtil(1680): hconnection-0xfe04c6-0x13de46b5f8b0007 Retrieved 43 byte(s) of data from znode /hbase/meta-region-server; data=asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:15:05,937 DEBUG [Thread-214] client.MetaScanner(251): Current INFO from scan results = {NAME => 'testLogSplittingAfterMasterRecoveryDueToZKExpiry,,1365336889784.fb4182aef4ce07f011871ae0a083aee0.', STARTKEY => '', ENDKEY => '1', ENCODED => fb4182aef4ce07f011871ae0a083aee0,} 2013-04-07 12:15:05,937 DEBUG [Thread-214] client.MetaScanner(198): Scanning .META. starting at row=testLogSplittingAfterMasterRecoveryDueToZKExpiry,,00000000000000 for max=10 rows using hconnection-0xfe04c6 2013-04-07 12:15:05,938 DEBUG [Thread-214] zookeeper.ZKUtil(1680): hconnection-0xfe04c6-0x13de46b5f8b0007 Retrieved 43 byte(s) of data from znode /hbase/meta-region-server; data=asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:15:05,942 DEBUG [Thread-214] client.MetaScanner(251): Current INFO from scan results = {NAME => 'testLogSplittingAfterMasterRecoveryDueToZKExpiry,,1365336889784.fb4182aef4ce07f011871ae0a083aee0.', STARTKEY => '', ENDKEY => '1', ENCODED => fb4182aef4ce07f011871ae0a083aee0,} 2013-04-07 12:15:05,943 DEBUG [Thread-214] client.HConnectionManager$HConnectionImplementation(1433): Cached location for testLogSplittingAfterMasterRecoveryDueToZKExpiry,,1365336889784.fb4182aef4ce07f011871ae0a083aee0. is asf001.sp2.ygridcore.net:60884 2013-04-07 12:15:05,943 DEBUG [Thread-214] client.MetaScanner(251): Current INFO from scan results = {NAME => 'testLogSplittingAfterMasterRecoveryDueToZKExpiry,1,1365336889784.43207168ac66dce85cf2a4f916007450.', STARTKEY => '1', ENDKEY => '2', ENCODED => 43207168ac66dce85cf2a4f916007450,} 2013-04-07 12:15:05,943 DEBUG [Thread-214] client.MetaScanner(251): Current INFO from scan results = {NAME => 'testLogSplittingAfterMasterRecoveryDueToZKExpiry,2,1365336889784.55884b67b9203a71bdf8c0e3de70452d.', STARTKEY => '2', ENDKEY => '3', ENCODED => 55884b67b9203a71bdf8c0e3de70452d,} 2013-04-07 12:15:05,944 DEBUG [Thread-214] client.MetaScanner(251): Current INFO from scan results = {NAME => 'testLogSplittingAfterMasterRecoveryDueToZKExpiry,3,1365336889784.4e9a3d63237d29e6bf55bf2360834cbe.', STARTKEY => '3', ENDKEY => '4', ENCODED => 4e9a3d63237d29e6bf55bf2360834cbe,} 2013-04-07 12:15:05,944 DEBUG [Thread-214] client.MetaScanner(251): Current INFO from scan results = {NAME => 'testLogSplittingAfterMasterRecoveryDueToZKExpiry,4,1365336889784.567d8ef20f90f3c1b8a889d22a1c3948.', STARTKEY => '4', ENDKEY => '5', ENCODED => 567d8ef20f90f3c1b8a889d22a1c3948,} 2013-04-07 12:15:05,944 DEBUG [Thread-214] client.MetaScanner(251): Current INFO from scan results = {NAME => 'testLogSplittingAfterMasterRecoveryDueToZKExpiry,5,1365336889784.5403a557a6168e5f3c5141eb494f6088.', STARTKEY => '5', ENDKEY => '', ENCODED => 5403a557a6168e5f3c5141eb494f6088,} 2013-04-07 12:15:13,935 DEBUG [IPC Reader 4 on port 60884] regionserver.QosFunction(169): Marking normal priority after getting exception=org.apache.hadoop.hbase.exceptions.NotServingRegionException: Region is not online: fb4182aef4ce07f011871ae0a083aee0 2013-04-07 12:15:13,935 DEBUG [IPC Reader 4 on port 60884] regionserver.QosFunction(169): Marking normal priority after getting exception=org.apache.hadoop.hbase.exceptions.NotServingRegionException: Region is not online: fb4182aef4ce07f011871ae0a083aee0 2013-04-07 12:15:13,938 DEBUG [Thread-214] zookeeper.ZKUtil(1680): hconnection-0xfe04c6-0x13de46b5f8b0007 Retrieved 43 byte(s) of data from znode /hbase/meta-region-server; data=asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:15:13,939 DEBUG [Thread-214] zookeeper.ZKUtil(1680): hconnection-0xfe04c6-0x13de46b5f8b0007 Retrieved 43 byte(s) of data from znode /hbase/meta-region-server; data=asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:15:13,940 DEBUG [Thread-214] zookeeper.ZKUtil(1680): hconnection-0xfe04c6-0x13de46b5f8b0007 Retrieved 43 byte(s) of data from znode /hbase/meta-region-server; data=asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:15:13,942 DEBUG [Thread-214] client.MetaScanner(251): Current INFO from scan results = {NAME => 'testLogSplittingAfterMasterRecoveryDueToZKExpiry,,1365336889784.fb4182aef4ce07f011871ae0a083aee0.', STARTKEY => '', ENDKEY => '1', ENCODED => fb4182aef4ce07f011871ae0a083aee0,} 2013-04-07 12:15:13,942 DEBUG [Thread-214] client.MetaScanner(198): Scanning .META. starting at row=testLogSplittingAfterMasterRecoveryDueToZKExpiry,,00000000000000 for max=10 rows using hconnection-0xfe04c6 2013-04-07 12:15:13,943 DEBUG [Thread-214] zookeeper.ZKUtil(1680): hconnection-0xfe04c6-0x13de46b5f8b0007 Retrieved 43 byte(s) of data from znode /hbase/meta-region-server; data=asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:15:13,946 DEBUG [Thread-214] client.MetaScanner(251): Current INFO from scan results = {NAME => 'testLogSplittingAfterMasterRecoveryDueToZKExpiry,,1365336889784.fb4182aef4ce07f011871ae0a083aee0.', STARTKEY => '', ENDKEY => '1', ENCODED => fb4182aef4ce07f011871ae0a083aee0,} 2013-04-07 12:15:13,947 DEBUG [Thread-214] client.HConnectionManager$HConnectionImplementation(1433): Cached location for testLogSplittingAfterMasterRecoveryDueToZKExpiry,,1365336889784.fb4182aef4ce07f011871ae0a083aee0. is asf001.sp2.ygridcore.net:60884 2013-04-07 12:15:13,947 DEBUG [Thread-214] client.MetaScanner(251): Current INFO from scan results = {NAME => 'testLogSplittingAfterMasterRecoveryDueToZKExpiry,1,1365336889784.43207168ac66dce85cf2a4f916007450.', STARTKEY => '1', ENDKEY => '2', ENCODED => 43207168ac66dce85cf2a4f916007450,} 2013-04-07 12:15:13,947 DEBUG [Thread-214] client.MetaScanner(251): Current INFO from scan results = {NAME => 'testLogSplittingAfterMasterRecoveryDueToZKExpiry,2,1365336889784.55884b67b9203a71bdf8c0e3de70452d.', STARTKEY => '2', ENDKEY => '3', ENCODED => 55884b67b9203a71bdf8c0e3de70452d,} 2013-04-07 12:15:13,947 DEBUG [Thread-214] client.MetaScanner(251): Current INFO from scan results = {NAME => 'testLogSplittingAfterMasterRecoveryDueToZKExpiry,3,1365336889784.4e9a3d63237d29e6bf55bf2360834cbe.', STARTKEY => '3', ENDKEY => '4', ENCODED => 4e9a3d63237d29e6bf55bf2360834cbe,} 2013-04-07 12:15:13,948 DEBUG [Thread-214] client.MetaScanner(251): Current INFO from scan results = {NAME => 'testLogSplittingAfterMasterRecoveryDueToZKExpiry,4,1365336889784.567d8ef20f90f3c1b8a889d22a1c3948.', STARTKEY => '4', ENDKEY => '5', ENCODED => 567d8ef20f90f3c1b8a889d22a1c3948,} 2013-04-07 12:15:13,948 DEBUG [Thread-214] client.MetaScanner(251): Current INFO from scan results = {NAME => 'testLogSplittingAfterMasterRecoveryDueToZKExpiry,5,1365336889784.5403a557a6168e5f3c5141eb494f6088.', STARTKEY => '5', ENDKEY => '', ENCODED => 5403a557a6168e5f3c5141eb494f6088,} 2013-04-07 12:15:30,092 DEBUG [IPC Reader 7 on port 60884] regionserver.QosFunction(169): Marking normal priority after getting exception=org.apache.hadoop.hbase.exceptions.NotServingRegionException: Region is not online: fb4182aef4ce07f011871ae0a083aee0 2013-04-07 12:15:30,092 DEBUG [IPC Reader 7 on port 60884] regionserver.QosFunction(169): Marking normal priority after getting exception=org.apache.hadoop.hbase.exceptions.NotServingRegionException: Region is not online: fb4182aef4ce07f011871ae0a083aee0 2013-04-07 12:15:30,095 DEBUG [Thread-214] zookeeper.ZKUtil(1680): hconnection-0xfe04c6-0x13de46b5f8b0007 Retrieved 43 byte(s) of data from znode /hbase/meta-region-server; data=asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:15:30,097 DEBUG [Thread-214] zookeeper.ZKUtil(1680): hconnection-0xfe04c6-0x13de46b5f8b0007 Retrieved 43 byte(s) of data from znode /hbase/meta-region-server; data=asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:15:30,098 DEBUG [Thread-214] zookeeper.ZKUtil(1680): hconnection-0xfe04c6-0x13de46b5f8b0007 Retrieved 43 byte(s) of data from znode /hbase/meta-region-server; data=asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:15:30,100 DEBUG [Thread-214] client.MetaScanner(251): Current INFO from scan results = {NAME => 'testLogSplittingAfterMasterRecoveryDueToZKExpiry,,1365336889784.fb4182aef4ce07f011871ae0a083aee0.', STARTKEY => '', ENDKEY => '1', ENCODED => fb4182aef4ce07f011871ae0a083aee0,} 2013-04-07 12:15:30,100 DEBUG [Thread-214] client.MetaScanner(198): Scanning .META. starting at row=testLogSplittingAfterMasterRecoveryDueToZKExpiry,,00000000000000 for max=10 rows using hconnection-0xfe04c6 2013-04-07 12:15:30,101 DEBUG [Thread-214] zookeeper.ZKUtil(1680): hconnection-0xfe04c6-0x13de46b5f8b0007 Retrieved 43 byte(s) of data from znode /hbase/meta-region-server; data=asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:15:30,106 DEBUG [Thread-214] client.MetaScanner(251): Current INFO from scan results = {NAME => 'testLogSplittingAfterMasterRecoveryDueToZKExpiry,,1365336889784.fb4182aef4ce07f011871ae0a083aee0.', STARTKEY => '', ENDKEY => '1', ENCODED => fb4182aef4ce07f011871ae0a083aee0,} 2013-04-07 12:15:30,106 DEBUG [Thread-214] client.HConnectionManager$HConnectionImplementation(1433): Cached location for testLogSplittingAfterMasterRecoveryDueToZKExpiry,,1365336889784.fb4182aef4ce07f011871ae0a083aee0. is asf001.sp2.ygridcore.net:60884 2013-04-07 12:15:30,107 DEBUG [Thread-214] client.MetaScanner(251): Current INFO from scan results = {NAME => 'testLogSplittingAfterMasterRecoveryDueToZKExpiry,1,1365336889784.43207168ac66dce85cf2a4f916007450.', STARTKEY => '1', ENDKEY => '2', ENCODED => 43207168ac66dce85cf2a4f916007450,} 2013-04-07 12:15:30,107 DEBUG [Thread-214] client.MetaScanner(251): Current INFO from scan results = {NAME => 'testLogSplittingAfterMasterRecoveryDueToZKExpiry,2,1365336889784.55884b67b9203a71bdf8c0e3de70452d.', STARTKEY => '2', ENDKEY => '3', ENCODED => 55884b67b9203a71bdf8c0e3de70452d,} 2013-04-07 12:15:30,107 DEBUG [Thread-214] client.MetaScanner(251): Current INFO from scan results = {NAME => 'testLogSplittingAfterMasterRecoveryDueToZKExpiry,3,1365336889784.4e9a3d63237d29e6bf55bf2360834cbe.', STARTKEY => '3', ENDKEY => '4', ENCODED => 4e9a3d63237d29e6bf55bf2360834cbe,} 2013-04-07 12:15:30,108 DEBUG [Thread-214] client.MetaScanner(251): Current INFO from scan results = {NAME => 'testLogSplittingAfterMasterRecoveryDueToZKExpiry,4,1365336889784.567d8ef20f90f3c1b8a889d22a1c3948.', STARTKEY => '4', ENDKEY => '5', ENCODED => 567d8ef20f90f3c1b8a889d22a1c3948,} 2013-04-07 12:15:30,108 DEBUG [Thread-214] client.MetaScanner(251): Current INFO from scan results = {NAME => 'testLogSplittingAfterMasterRecoveryDueToZKExpiry,5,1365336889784.5403a557a6168e5f3c5141eb494f6088.', STARTKEY => '5', ENDKEY => '', ENCODED => 5403a557a6168e5f3c5141eb494f6088,} 2013-04-07 12:16:02,233 DEBUG [IPC Reader 8 on port 60884] regionserver.QosFunction(169): Marking normal priority after getting exception=org.apache.hadoop.hbase.exceptions.NotServingRegionException: Region is not online: fb4182aef4ce07f011871ae0a083aee0 2013-04-07 12:16:02,233 DEBUG [IPC Reader 8 on port 60884] regionserver.QosFunction(169): Marking normal priority after getting exception=org.apache.hadoop.hbase.exceptions.NotServingRegionException: Region is not online: fb4182aef4ce07f011871ae0a083aee0 2013-04-07 12:16:02,236 DEBUG [Thread-214] zookeeper.ZKUtil(1680): hconnection-0xfe04c6-0x13de46b5f8b0007 Retrieved 43 byte(s) of data from znode /hbase/meta-region-server; data=asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:16:02,238 DEBUG [Thread-214] zookeeper.ZKUtil(1680): hconnection-0xfe04c6-0x13de46b5f8b0007 Retrieved 43 byte(s) of data from znode /hbase/meta-region-server; data=asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:16:02,238 DEBUG [Thread-214] zookeeper.ZKUtil(1680): hconnection-0xfe04c6-0x13de46b5f8b0007 Retrieved 43 byte(s) of data from znode /hbase/meta-region-server; data=asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:16:02,241 DEBUG [Thread-214] client.MetaScanner(251): Current INFO from scan results = {NAME => 'testLogSplittingAfterMasterRecoveryDueToZKExpiry,,1365336889784.fb4182aef4ce07f011871ae0a083aee0.', STARTKEY => '', ENDKEY => '1', ENCODED => fb4182aef4ce07f011871ae0a083aee0,} 2013-04-07 12:16:02,241 DEBUG [Thread-214] client.MetaScanner(198): Scanning .META. starting at row=testLogSplittingAfterMasterRecoveryDueToZKExpiry,,00000000000000 for max=10 rows using hconnection-0xfe04c6 2013-04-07 12:16:02,242 DEBUG [Thread-214] zookeeper.ZKUtil(1680): hconnection-0xfe04c6-0x13de46b5f8b0007 Retrieved 43 byte(s) of data from znode /hbase/meta-region-server; data=asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:16:02,248 DEBUG [Thread-214] client.MetaScanner(251): Current INFO from scan results = {NAME => 'testLogSplittingAfterMasterRecoveryDueToZKExpiry,,1365336889784.fb4182aef4ce07f011871ae0a083aee0.', STARTKEY => '', ENDKEY => '1', ENCODED => fb4182aef4ce07f011871ae0a083aee0,} 2013-04-07 12:16:02,248 DEBUG [Thread-214] client.HConnectionManager$HConnectionImplementation(1433): Cached location for testLogSplittingAfterMasterRecoveryDueToZKExpiry,,1365336889784.fb4182aef4ce07f011871ae0a083aee0. is asf001.sp2.ygridcore.net:60884 2013-04-07 12:16:02,249 DEBUG [Thread-214] client.MetaScanner(251): Current INFO from scan results = {NAME => 'testLogSplittingAfterMasterRecoveryDueToZKExpiry,1,1365336889784.43207168ac66dce85cf2a4f916007450.', STARTKEY => '1', ENDKEY => '2', ENCODED => 43207168ac66dce85cf2a4f916007450,} 2013-04-07 12:16:02,249 DEBUG [Thread-214] client.MetaScanner(251): Current INFO from scan results = {NAME => 'testLogSplittingAfterMasterRecoveryDueToZKExpiry,2,1365336889784.55884b67b9203a71bdf8c0e3de70452d.', STARTKEY => '2', ENDKEY => '3', ENCODED => 55884b67b9203a71bdf8c0e3de70452d,} 2013-04-07 12:16:02,249 DEBUG [Thread-214] client.MetaScanner(251): Current INFO from scan results = {NAME => 'testLogSplittingAfterMasterRecoveryDueToZKExpiry,3,1365336889784.4e9a3d63237d29e6bf55bf2360834cbe.', STARTKEY => '3', ENDKEY => '4', ENCODED => 4e9a3d63237d29e6bf55bf2360834cbe,} 2013-04-07 12:16:02,249 DEBUG [Thread-214] client.MetaScanner(251): Current INFO from scan results = {NAME => 'testLogSplittingAfterMasterRecoveryDueToZKExpiry,4,1365336889784.567d8ef20f90f3c1b8a889d22a1c3948.', STARTKEY => '4', ENDKEY => '5', ENCODED => 567d8ef20f90f3c1b8a889d22a1c3948,} 2013-04-07 12:16:02,250 DEBUG [Thread-214] client.MetaScanner(251): Current INFO from scan results = {NAME => 'testLogSplittingAfterMasterRecoveryDueToZKExpiry,5,1365336889784.5403a557a6168e5f3c5141eb494f6088.', STARTKEY => '5', ENDKEY => '', ENCODED => 5403a557a6168e5f3c5141eb494f6088,} 2013-04-07 12:16:02,252 DEBUG [IPC Reader 8 on port 60884] regionserver.QosFunction(169): Marking normal priority after getting exception=org.apache.hadoop.hbase.exceptions.NotServingRegionException: Region is not online: fb4182aef4ce07f011871ae0a083aee0 2013-04-07 12:16:02,252 DEBUG [IPC Reader 8 on port 60884] regionserver.QosFunction(169): Marking normal priority after getting exception=org.apache.hadoop.hbase.exceptions.NotServingRegionException: Region is not online: fb4182aef4ce07f011871ae0a083aee0 2013-04-07 12:16:02,280 INFO [pool-1-thread-1] hbase.ResourceChecker(171): after: TestZooKeeper#testLogSplittingAfterMasterRecoveryDueToZKExpiry Thread=225 (was 213) - Thread LEAK? -, OpenFileDescriptor=357 (was 343) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=306 (was 320), ProcessCount=175 (was 175), AvailableMemoryMB=7199 (was 7064) - AvailableMemoryMB LEAK? -, ConnectionCount=5 (was 4) - ConnectionCount LEAK? - 2013-04-07 12:16:03,018 INFO [PRI IPC Server handler 1 on 43802] regionserver.HRegionServer(3346): Received request to open region: testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,c,1365336962398.4432e800611c00a4ae168862a273fc70. on asf001.sp2.ygridcore.net,43802,1365336889418 2013-04-07 12:16:03,018 INFO [asf001.sp2.ygridcore.net,33796,1365336878036-GeneralBulkAssigner-1] master.AssignmentManager(1482): asf001.sp2.ygridcore.net,49518,1365336962323 unassigned znodes=4 of total=4 2013-04-07 12:16:03,019 INFO [asf001.sp2.ygridcore.net,33796,1365336878036-GeneralBulkAssigner-1] master.RegionStates(264): Region {NAME => 'testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,d,1365336962398.e99798d2c4d566ddb063efe03842457f.', STARTKEY => 'd', ENDKEY => 'e', ENCODED => e99798d2c4d566ddb063efe03842457f,} transitioned from {testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,d,1365336962398.e99798d2c4d566ddb063efe03842457f. state=OFFLINE, ts=1365336962992, server=null} to {testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,d,1365336962398.e99798d2c4d566ddb063efe03842457f. state=PENDING_OPEN, ts=1365336963019, server=asf001.sp2.ygridcore.net,49518,1365336962323} 2013-04-07 12:16:03,019 INFO [asf001.sp2.ygridcore.net,33796,1365336878036-GeneralBulkAssigner-1] master.RegionStates(264): Region {NAME => 'testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,g,1365336962398.682475e1358b7306bd3406e9330263a4.', STARTKEY => 'g', ENDKEY => 'h', ENCODED => 682475e1358b7306bd3406e9330263a4,} transitioned from {testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,g,1365336962398.682475e1358b7306bd3406e9330263a4. state=OFFLINE, ts=1365336962994, server=null} to {testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,g,1365336962398.682475e1358b7306bd3406e9330263a4. state=PENDING_OPEN, ts=1365336963019, server=asf001.sp2.ygridcore.net,49518,1365336962323} 2013-04-07 12:16:03,019 INFO [asf001.sp2.ygridcore.net,33796,1365336878036-GeneralBulkAssigner-1] master.RegionStates(264): Region {NAME => 'testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,a,1365336962398.b1b340dff1e06fd6cb0b0a9c64964551.', STARTKEY => 'a', ENDKEY => 'b', ENCODED => b1b340dff1e06fd6cb0b0a9c64964551,} transitioned from {testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,a,1365336962398.b1b340dff1e06fd6cb0b0a9c64964551. state=OFFLINE, ts=1365336962995, server=null} to {testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,a,1365336962398.b1b340dff1e06fd6cb0b0a9c64964551. state=PENDING_OPEN, ts=1365336963019, server=asf001.sp2.ygridcore.net,49518,1365336962323} 2013-04-07 12:16:03,019 INFO [asf001.sp2.ygridcore.net,33796,1365336878036-GeneralBulkAssigner-1] master.RegionStates(264): Region {NAME => 'testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,j,1365336962398.38f0eb146fb473b19541deaada3419b9.', STARTKEY => 'j', ENDKEY => '', ENCODED => 38f0eb146fb473b19541deaada3419b9,} transitioned from {testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,j,1365336962398.38f0eb146fb473b19541deaada3419b9. state=OFFLINE, ts=1365336962996, server=null} to {testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,j,1365336962398.38f0eb146fb473b19541deaada3419b9. state=PENDING_OPEN, ts=1365336963019, server=asf001.sp2.ygridcore.net,49518,1365336962323} 2013-04-07 12:16:03,019 DEBUG [asf001.sp2.ygridcore.net,33796,1365336878036-GeneralBulkAssigner-1] master.ServerManager(713): New connection to asf001.sp2.ygridcore.net,49518,1365336962323 2013-04-07 12:16:03,033 INFO [PRI IPC Server handler 1 on 43802] regionserver.HRegionServer(3346): Received request to open region: testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,,1365336962397.d4e129e51e67d798e17083623fdde133. on asf001.sp2.ygridcore.net,43802,1365336889418 2013-04-07 12:16:03,034 INFO [PRI IPC Server handler 1 on 43802] regionserver.HRegionServer(3346): Received request to open region: testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,h,1365336962398.1ce6d62427641e2a0e135d3d27b54b5d. on asf001.sp2.ygridcore.net,43802,1365336889418 2013-04-07 12:16:03,035 DEBUG [RS_OPEN_REGION-asf001.sp2.ygridcore.net,43802,1365336889418-0] zookeeper.ZKAssign(721): regionserver:43802-0x13de46b5f8b000f Attempting to transition node 4432e800611c00a4ae168862a273fc70 from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING 2013-04-07 12:16:03,035 DEBUG [RS_OPEN_REGION-asf001.sp2.ygridcore.net,43802,1365336889418-1] zookeeper.ZKAssign(721): regionserver:43802-0x13de46b5f8b000f Attempting to transition node d4e129e51e67d798e17083623fdde133 from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING 2013-04-07 12:16:03,035 DEBUG [RS_OPEN_REGION-asf001.sp2.ygridcore.net,43802,1365336889418-2] zookeeper.ZKAssign(721): regionserver:43802-0x13de46b5f8b000f Attempting to transition node 1ce6d62427641e2a0e135d3d27b54b5d from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING 2013-04-07 12:16:03,038 DEBUG [asf001.sp2.ygridcore.net,33796,1365336878036-GeneralBulkAssigner-0] master.AssignmentManager(1600): Bulk assigning done for asf001.sp2.ygridcore.net,43802,1365336889418 2013-04-07 12:16:03,040 DEBUG [RS_OPEN_REGION-asf001.sp2.ygridcore.net,43802,1365336889418-0] zookeeper.ZKUtil(1680): regionserver:43802-0x13de46b5f8b000f Retrieved 156 byte(s) of data from znode /hbase/region-in-transition/4432e800611c00a4ae168862a273fc70; data=region=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,c,1365336962398.4432e800611c00a4ae168862a273fc70., state=M_ZK_REGION_OFFLINE, servername=asf001.sp2.ygridcore.net,43802,1365336889418, createTime=1365336962993, payload.length=0 2013-04-07 12:16:03,041 DEBUG [RS_OPEN_REGION-asf001.sp2.ygridcore.net,60884,1365336878389-0] zookeeper.ZKAssign(721): regionserver:60884-0x13de46b5f8b0001 Attempting to transition node dbb82f7f432d6664327da0b8f15b7d76 from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING 2013-04-07 12:16:03,041 DEBUG [asf001.sp2.ygridcore.net,33796,1365336878036-GeneralBulkAssigner-2] master.AssignmentManager(1600): Bulk assigning done for asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:16:03,044 DEBUG [RS_OPEN_REGION-asf001.sp2.ygridcore.net,43802,1365336889418-2] zookeeper.ZKUtil(1680): regionserver:43802-0x13de46b5f8b000f Retrieved 156 byte(s) of data from znode /hbase/region-in-transition/1ce6d62427641e2a0e135d3d27b54b5d; data=region=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,h,1365336962398.1ce6d62427641e2a0e135d3d27b54b5d., state=M_ZK_REGION_OFFLINE, servername=asf001.sp2.ygridcore.net,43802,1365336889418, createTime=1365336962995, payload.length=0 2013-04-07 12:16:03,049 DEBUG [RS_OPEN_REGION-asf001.sp2.ygridcore.net,43802,1365336889418-1] zookeeper.ZKUtil(1680): regionserver:43802-0x13de46b5f8b000f Retrieved 155 byte(s) of data from znode /hbase/region-in-transition/d4e129e51e67d798e17083623fdde133; data=region=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,,1365336962397.d4e129e51e67d798e17083623fdde133., state=M_ZK_REGION_OFFLINE, servername=asf001.sp2.ygridcore.net,43802,1365336889418, createTime=1365336962994, payload.length=0 2013-04-07 12:16:03,053 DEBUG [RS_OPEN_REGION-asf001.sp2.ygridcore.net,43802,1365336889418-0] zookeeper.ZKAssign(784): regionserver:43802-0x13de46b5f8b000f Successfully transitioned node 4432e800611c00a4ae168862a273fc70 from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING 2013-04-07 12:16:03,053 INFO [RS_OPEN_REGION-asf001.sp2.ygridcore.net,43802,1365336889418-0] regionserver.HRegion(4023): HRegion.openHRegion Region name ==testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,c,1365336962398.4432e800611c00a4ae168862a273fc70. 2013-04-07 12:16:03,054 DEBUG [RS_OPEN_REGION-asf001.sp2.ygridcore.net,43802,1365336889418-0] regionserver.HRegion(4025): Opening region: {NAME => 'testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,c,1365336962398.4432e800611c00a4ae168862a273fc70.', STARTKEY => 'c', ENDKEY => 'd', ENCODED => 4432e800611c00a4ae168862a273fc70,} 2013-04-07 12:16:03,056 DEBUG [RS_OPEN_REGION-asf001.sp2.ygridcore.net,43802,1365336889418-0] regionserver.MetricsRegionSourceImpl(54): Creating new MetricsRegionSourceImpl for table testRegionAssignmentAfterMasterRecoveryDueToZKExpiry 4432e800611c00a4ae168862a273fc70 2013-04-07 12:16:03,057 DEBUG [RS_OPEN_REGION-asf001.sp2.ygridcore.net,43802,1365336889418-2] zookeeper.ZKAssign(784): regionserver:43802-0x13de46b5f8b000f Successfully transitioned node 1ce6d62427641e2a0e135d3d27b54b5d from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING 2013-04-07 12:16:03,057 DEBUG [RS_OPEN_REGION-asf001.sp2.ygridcore.net,43802,1365336889418-0] regionserver.HRegion(496): Instantiated testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,c,1365336962398.4432e800611c00a4ae168862a273fc70. 2013-04-07 12:16:03,057 DEBUG [RS_OPEN_REGION-asf001.sp2.ygridcore.net,60884,1365336878389-0] zookeeper.ZKAssign(784): regionserver:60884-0x13de46b5f8b0001 Successfully transitioned node dbb82f7f432d6664327da0b8f15b7d76 from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING 2013-04-07 12:16:03,057 INFO [RS_OPEN_REGION-asf001.sp2.ygridcore.net,43802,1365336889418-2] regionserver.HRegion(4023): HRegion.openHRegion Region name ==testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,h,1365336962398.1ce6d62427641e2a0e135d3d27b54b5d. 2013-04-07 12:16:03,057 DEBUG [RS_OPEN_REGION-asf001.sp2.ygridcore.net,43802,1365336889418-2] regionserver.HRegion(4025): Opening region: {NAME => 'testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,h,1365336962398.1ce6d62427641e2a0e135d3d27b54b5d.', STARTKEY => 'h', ENDKEY => 'i', ENCODED => 1ce6d62427641e2a0e135d3d27b54b5d,} 2013-04-07 12:16:03,057 INFO [RS_OPEN_REGION-asf001.sp2.ygridcore.net,60884,1365336878389-0] regionserver.HRegion(4023): HRegion.openHRegion Region name ==testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,e,1365336962398.dbb82f7f432d6664327da0b8f15b7d76. 2013-04-07 12:16:03,057 DEBUG [RS_OPEN_REGION-asf001.sp2.ygridcore.net,60884,1365336878389-0] regionserver.HRegion(4025): Opening region: {NAME => 'testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,e,1365336962398.dbb82f7f432d6664327da0b8f15b7d76.', STARTKEY => 'e', ENDKEY => 'f', ENCODED => dbb82f7f432d6664327da0b8f15b7d76,} 2013-04-07 12:16:03,058 DEBUG [RS_OPEN_REGION-asf001.sp2.ygridcore.net,60884,1365336878389-2] zookeeper.ZKUtil(1680): regionserver:60884-0x13de46b5f8b0001 Retrieved 156 byte(s) of data from znode /hbase/region-in-transition/385e5846630b0cb4493758961a2c2457; data=region=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,i,1365336962398.385e5846630b0cb4493758961a2c2457., state=M_ZK_REGION_OFFLINE, servername=asf001.sp2.ygridcore.net,60884,1365336878389, createTime=1365336962995, payload.length=0 2013-04-07 12:16:03,059 DEBUG [RS_OPEN_REGION-asf001.sp2.ygridcore.net,43802,1365336889418-2] regionserver.MetricsRegionSourceImpl(54): Creating new MetricsRegionSourceImpl for table testRegionAssignmentAfterMasterRecoveryDueToZKExpiry 1ce6d62427641e2a0e135d3d27b54b5d 2013-04-07 12:16:03,059 DEBUG [RS_OPEN_REGION-asf001.sp2.ygridcore.net,43802,1365336889418-2] regionserver.HRegion(496): Instantiated testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,h,1365336962398.1ce6d62427641e2a0e135d3d27b54b5d. 2013-04-07 12:16:03,061 DEBUG [RS_OPEN_REGION-asf001.sp2.ygridcore.net,43802,1365336889418-1] zookeeper.ZKAssign(784): regionserver:43802-0x13de46b5f8b000f Successfully transitioned node d4e129e51e67d798e17083623fdde133 from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING 2013-04-07 12:16:03,062 INFO [RS_OPEN_REGION-asf001.sp2.ygridcore.net,43802,1365336889418-1] regionserver.HRegion(4023): HRegion.openHRegion Region name ==testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,,1365336962397.d4e129e51e67d798e17083623fdde133. 2013-04-07 12:16:03,062 DEBUG [RS_OPEN_REGION-asf001.sp2.ygridcore.net,43802,1365336889418-1] regionserver.HRegion(4025): Opening region: {NAME => 'testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,,1365336962397.d4e129e51e67d798e17083623fdde133.', STARTKEY => '', ENDKEY => 'a', ENCODED => d4e129e51e67d798e17083623fdde133,} 2013-04-07 12:16:03,062 INFO [RS_OPEN_REGION-asf001.sp2.ygridcore.net,43802,1365336889418-0] regionserver.HRegion(586): Onlined testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,c,1365336962398.4432e800611c00a4ae168862a273fc70.; next sequenceid=0 2013-04-07 12:16:03,064 DEBUG [RS_OPEN_REGION-asf001.sp2.ygridcore.net,43802,1365336889418-0] zookeeper.ZKAssign(721): regionserver:43802-0x13de46b5f8b000f Attempting to transition node 4432e800611c00a4ae168862a273fc70 from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENING 2013-04-07 12:16:03,065 DEBUG [RS_OPEN_REGION-asf001.sp2.ygridcore.net,43802,1365336889418-1] regionserver.MetricsRegionSourceImpl(54): Creating new MetricsRegionSourceImpl for table testRegionAssignmentAfterMasterRecoveryDueToZKExpiry d4e129e51e67d798e17083623fdde133 2013-04-07 12:16:03,065 DEBUG [RS_OPEN_REGION-asf001.sp2.ygridcore.net,43802,1365336889418-1] regionserver.HRegion(496): Instantiated testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,,1365336962397.d4e129e51e67d798e17083623fdde133. 2013-04-07 12:16:03,070 DEBUG [RS_OPEN_REGION-asf001.sp2.ygridcore.net,43802,1365336889418-0] zookeeper.ZKUtil(1680): regionserver:43802-0x13de46b5f8b000f Retrieved 156 byte(s) of data from znode /hbase/region-in-transition/4432e800611c00a4ae168862a273fc70; data=region=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,c,1365336962398.4432e800611c00a4ae168862a273fc70., state=RS_ZK_REGION_OPENING, servername=asf001.sp2.ygridcore.net,43802,1365336889418, createTime=1365336963040, payload.length=0 2013-04-07 12:16:03,080 DEBUG [RS_OPEN_REGION-asf001.sp2.ygridcore.net,43802,1365336889418-0] zookeeper.ZKAssign(784): regionserver:43802-0x13de46b5f8b000f Successfully transitioned node 4432e800611c00a4ae168862a273fc70 from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENING 2013-04-07 12:16:03,080 INFO [RS_OPEN_REGION-asf001.sp2.ygridcore.net,60884,1365336878389-0] regionserver.HRegion(586): Onlined testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,e,1365336962398.dbb82f7f432d6664327da0b8f15b7d76.; next sequenceid=0 2013-04-07 12:16:03,080 DEBUG [RS_OPEN_REGION-asf001.sp2.ygridcore.net,60884,1365336878389-0] zookeeper.ZKAssign(721): regionserver:60884-0x13de46b5f8b0001 Attempting to transition node dbb82f7f432d6664327da0b8f15b7d76 from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENING 2013-04-07 12:16:03,081 INFO [RS_OPEN_REGION-asf001.sp2.ygridcore.net,43802,1365336889418-2] regionserver.HRegion(586): Onlined testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,h,1365336962398.1ce6d62427641e2a0e135d3d27b54b5d.; next sequenceid=0 2013-04-07 12:16:03,081 DEBUG [RS_OPEN_REGION-asf001.sp2.ygridcore.net,43802,1365336889418-2] zookeeper.ZKAssign(721): regionserver:43802-0x13de46b5f8b000f Attempting to transition node 1ce6d62427641e2a0e135d3d27b54b5d from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENING 2013-04-07 12:16:03,081 INFO [RS_OPEN_REGION-asf001.sp2.ygridcore.net,43802,1365336889418-1] regionserver.HRegion(586): Onlined testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,,1365336962397.d4e129e51e67d798e17083623fdde133.; next sequenceid=0 2013-04-07 12:16:03,081 INFO [PostOpenDeployTasks:4432e800611c00a4ae168862a273fc70] regionserver.HRegionServer(1563): Post open deploy tasks for region=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,c,1365336962398.4432e800611c00a4ae168862a273fc70. 2013-04-07 12:16:03,081 DEBUG [RS_OPEN_REGION-asf001.sp2.ygridcore.net,43802,1365336889418-1] zookeeper.ZKAssign(721): regionserver:43802-0x13de46b5f8b000f Attempting to transition node d4e129e51e67d798e17083623fdde133 from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENING 2013-04-07 12:16:03,086 DEBUG [RS_OPEN_REGION-asf001.sp2.ygridcore.net,43802,1365336889418-2] zookeeper.ZKUtil(1680): regionserver:43802-0x13de46b5f8b000f Retrieved 156 byte(s) of data from znode /hbase/region-in-transition/1ce6d62427641e2a0e135d3d27b54b5d; data=region=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,h,1365336962398.1ce6d62427641e2a0e135d3d27b54b5d., state=RS_ZK_REGION_OPENING, servername=asf001.sp2.ygridcore.net,43802,1365336889418, createTime=1365336963044, payload.length=0 2013-04-07 12:16:03,088 DEBUG [RS_OPEN_REGION-asf001.sp2.ygridcore.net,43802,1365336889418-1] zookeeper.ZKUtil(1680): regionserver:43802-0x13de46b5f8b000f Retrieved 155 byte(s) of data from znode /hbase/region-in-transition/d4e129e51e67d798e17083623fdde133; data=region=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,,1365336962397.d4e129e51e67d798e17083623fdde133., state=RS_ZK_REGION_OPENING, servername=asf001.sp2.ygridcore.net,43802,1365336889418, createTime=1365336963049, payload.length=0 2013-04-07 12:16:03,089 DEBUG [PostOpenDeployTasks:4432e800611c00a4ae168862a273fc70] zookeeper.ZKUtil(1680): hconnection-0x1c9f8fb-0x13de46b5f8b0010 Retrieved 43 byte(s) of data from znode /hbase/meta-region-server; data=asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:16:03,092 DEBUG [RS_OPEN_REGION-asf001.sp2.ygridcore.net,43802,1365336889418-2] zookeeper.ZKAssign(784): regionserver:43802-0x13de46b5f8b000f Successfully transitioned node 1ce6d62427641e2a0e135d3d27b54b5d from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENING 2013-04-07 12:16:03,093 DEBUG [RS_OPEN_REGION-asf001.sp2.ygridcore.net,43802,1365336889418-1] zookeeper.ZKAssign(784): regionserver:43802-0x13de46b5f8b000f Successfully transitioned node d4e129e51e67d798e17083623fdde133 from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENING 2013-04-07 12:16:03,093 INFO [PostOpenDeployTasks:d4e129e51e67d798e17083623fdde133] regionserver.HRegionServer(1563): Post open deploy tasks for region=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,,1365336962397.d4e129e51e67d798e17083623fdde133. 2013-04-07 12:16:03,093 INFO [PostOpenDeployTasks:dbb82f7f432d6664327da0b8f15b7d76] regionserver.HRegionServer(1563): Post open deploy tasks for region=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,e,1365336962398.dbb82f7f432d6664327da0b8f15b7d76. 2013-04-07 12:16:03,093 INFO [PostOpenDeployTasks:1ce6d62427641e2a0e135d3d27b54b5d] regionserver.HRegionServer(1563): Post open deploy tasks for region=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,h,1365336962398.1ce6d62427641e2a0e135d3d27b54b5d. 2013-04-07 12:16:03,096 DEBUG [PostOpenDeployTasks:4432e800611c00a4ae168862a273fc70] zookeeper.ZKUtil(1680): hconnection-0x1c9f8fb-0x13de46b5f8b0010 Retrieved 43 byte(s) of data from znode /hbase/meta-region-server; data=asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:16:03,099 DEBUG [PostOpenDeployTasks:d4e129e51e67d798e17083623fdde133] zookeeper.ZKUtil(1680): hconnection-0x1c9f8fb-0x13de46b5f8b0010 Retrieved 43 byte(s) of data from znode /hbase/meta-region-server; data=asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:16:03,100 DEBUG [PostOpenDeployTasks:1ce6d62427641e2a0e135d3d27b54b5d] zookeeper.ZKUtil(1680): hconnection-0x1c9f8fb-0x13de46b5f8b0010 Retrieved 43 byte(s) of data from znode /hbase/meta-region-server; data=asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:16:03,102 INFO [PostOpenDeployTasks:4432e800611c00a4ae168862a273fc70] catalog.MetaEditor(432): Updated row testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,c,1365336962398.4432e800611c00a4ae168862a273fc70. with server=asf001.sp2.ygridcore.net,43802,1365336889418 2013-04-07 12:16:03,102 INFO [PostOpenDeployTasks:4432e800611c00a4ae168862a273fc70] regionserver.HRegionServer(1584): Done with post open deploy task for region=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,c,1365336962398.4432e800611c00a4ae168862a273fc70. 2013-04-07 12:16:03,103 DEBUG [PostOpenDeployTasks:d4e129e51e67d798e17083623fdde133] zookeeper.ZKUtil(1680): hconnection-0x1c9f8fb-0x13de46b5f8b0010 Retrieved 43 byte(s) of data from znode /hbase/meta-region-server; data=asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:16:03,105 DEBUG [RS_OPEN_REGION-asf001.sp2.ygridcore.net,43802,1365336889418-0] zookeeper.ZKAssign(721): regionserver:43802-0x13de46b5f8b000f Attempting to transition node 4432e800611c00a4ae168862a273fc70 from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENED 2013-04-07 12:16:03,108 DEBUG [PostOpenDeployTasks:1ce6d62427641e2a0e135d3d27b54b5d] zookeeper.ZKUtil(1680): hconnection-0x1c9f8fb-0x13de46b5f8b0010 Retrieved 43 byte(s) of data from znode /hbase/meta-region-server; data=asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:16:03,114 INFO [PostOpenDeployTasks:d4e129e51e67d798e17083623fdde133] catalog.MetaEditor(432): Updated row testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,,1365336962397.d4e129e51e67d798e17083623fdde133. with server=asf001.sp2.ygridcore.net,43802,1365336889418 2013-04-07 12:16:03,114 INFO [PostOpenDeployTasks:d4e129e51e67d798e17083623fdde133] regionserver.HRegionServer(1584): Done with post open deploy task for region=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,,1365336962397.d4e129e51e67d798e17083623fdde133. 2013-04-07 12:16:03,114 DEBUG [RS_OPEN_REGION-asf001.sp2.ygridcore.net,43802,1365336889418-1] zookeeper.ZKAssign(721): regionserver:43802-0x13de46b5f8b000f Attempting to transition node d4e129e51e67d798e17083623fdde133 from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENED 2013-04-07 12:16:03,114 DEBUG [PostOpenDeployTasks:dbb82f7f432d6664327da0b8f15b7d76] zookeeper.ZKUtil(1680): hconnection-0x43ad4a-0x13de46b5f8b0005 Retrieved 43 byte(s) of data from znode /hbase/meta-region-server; data=asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:16:03,133 DEBUG [RS_OPEN_REGION-asf001.sp2.ygridcore.net,43802,1365336889418-0] zookeeper.ZKUtil(1680): regionserver:43802-0x13de46b5f8b000f Retrieved 156 byte(s) of data from znode /hbase/region-in-transition/4432e800611c00a4ae168862a273fc70; data=region=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,c,1365336962398.4432e800611c00a4ae168862a273fc70., state=RS_ZK_REGION_OPENING, servername=asf001.sp2.ygridcore.net,43802,1365336889418, createTime=1365336963071, payload.length=0 2013-04-07 12:16:03,133 INFO [PostOpenDeployTasks:eff8376859f5e6ba3201ec0ec9536d0f] catalog.MetaEditor(432): Updated row testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,f,1365336962398.eff8376859f5e6ba3201ec0ec9536d0f. with server=asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:16:03,133 INFO [PostOpenDeployTasks:dbb82f7f432d6664327da0b8f15b7d76] catalog.MetaEditor(432): Updated row testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,e,1365336962398.dbb82f7f432d6664327da0b8f15b7d76. with server=asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:16:03,133 INFO [PostOpenDeployTasks:eff8376859f5e6ba3201ec0ec9536d0f] regionserver.HRegionServer(1584): Done with post open deploy task for region=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,f,1365336962398.eff8376859f5e6ba3201ec0ec9536d0f. 2013-04-07 12:16:03,133 INFO [PostOpenDeployTasks:dbb82f7f432d6664327da0b8f15b7d76] regionserver.HRegionServer(1584): Done with post open deploy task for region=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,e,1365336962398.dbb82f7f432d6664327da0b8f15b7d76. 2013-04-07 12:16:03,133 DEBUG [RS_OPEN_REGION-asf001.sp2.ygridcore.net,60884,1365336878389-1] zookeeper.ZKAssign(721): regionserver:60884-0x13de46b5f8b0001 Attempting to transition node eff8376859f5e6ba3201ec0ec9536d0f from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENED 2013-04-07 12:16:03,136 DEBUG [RS_OPEN_REGION-asf001.sp2.ygridcore.net,43802,1365336889418-1] zookeeper.ZKUtil(1680): regionserver:43802-0x13de46b5f8b000f Retrieved 155 byte(s) of data from znode /hbase/region-in-transition/d4e129e51e67d798e17083623fdde133; data=region=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,,1365336962397.d4e129e51e67d798e17083623fdde133., state=RS_ZK_REGION_OPENING, servername=asf001.sp2.ygridcore.net,43802,1365336889418, createTime=1365336963088, payload.length=0 2013-04-07 12:16:03,143 DEBUG [RS_OPEN_REGION-asf001.sp2.ygridcore.net,43802,1365336889418-1] zookeeper.ZKAssign(784): regionserver:43802-0x13de46b5f8b000f Successfully transitioned node d4e129e51e67d798e17083623fdde133 from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENED 2013-04-07 12:16:03,143 DEBUG [RS_OPEN_REGION-asf001.sp2.ygridcore.net,43802,1365336889418-1] handler.OpenRegionHandler(361): region transitioned to opened in zookeeper: {NAME => 'testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,,1365336962397.d4e129e51e67d798e17083623fdde133.', STARTKEY => '', ENDKEY => 'a', ENCODED => d4e129e51e67d798e17083623fdde133,}, server: asf001.sp2.ygridcore.net,43802,1365336889418 2013-04-07 12:16:03,143 DEBUG [RS_OPEN_REGION-asf001.sp2.ygridcore.net,43802,1365336889418-0] zookeeper.ZKAssign(784): regionserver:43802-0x13de46b5f8b000f Successfully transitioned node 4432e800611c00a4ae168862a273fc70 from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENED 2013-04-07 12:16:03,144 DEBUG [RS_OPEN_REGION-asf001.sp2.ygridcore.net,43802,1365336889418-0] handler.OpenRegionHandler(361): region transitioned to opened in zookeeper: {NAME => 'testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,c,1365336962398.4432e800611c00a4ae168862a273fc70.', STARTKEY => 'c', ENDKEY => 'd', ENCODED => 4432e800611c00a4ae168862a273fc70,}, server: asf001.sp2.ygridcore.net,43802,1365336889418 2013-04-07 12:16:03,144 DEBUG [RS_OPEN_REGION-asf001.sp2.ygridcore.net,43802,1365336889418-0] handler.OpenRegionHandler(168): Opened testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,c,1365336962398.4432e800611c00a4ae168862a273fc70. on server:asf001.sp2.ygridcore.net,43802,1365336889418 2013-04-07 12:16:03,144 DEBUG [RS_OPEN_REGION-asf001.sp2.ygridcore.net,43802,1365336889418-1] handler.OpenRegionHandler(168): Opened testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,,1365336962397.d4e129e51e67d798e17083623fdde133. on server:asf001.sp2.ygridcore.net,43802,1365336889418 2013-04-07 12:16:03,155 INFO [PostOpenDeployTasks:1ce6d62427641e2a0e135d3d27b54b5d] catalog.MetaEditor(432): Updated row testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,h,1365336962398.1ce6d62427641e2a0e135d3d27b54b5d. with server=asf001.sp2.ygridcore.net,43802,1365336889418 2013-04-07 12:16:03,156 INFO [PostOpenDeployTasks:1ce6d62427641e2a0e135d3d27b54b5d] regionserver.HRegionServer(1584): Done with post open deploy task for region=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,h,1365336962398.1ce6d62427641e2a0e135d3d27b54b5d. 2013-04-07 12:16:03,157 DEBUG [RS_OPEN_REGION-asf001.sp2.ygridcore.net,43802,1365336889418-2] zookeeper.ZKAssign(721): regionserver:43802-0x13de46b5f8b000f Attempting to transition node 1ce6d62427641e2a0e135d3d27b54b5d from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENED 2013-04-07 12:16:03,163 DEBUG [RS_OPEN_REGION-asf001.sp2.ygridcore.net,43802,1365336889418-2] zookeeper.ZKUtil(1680): regionserver:43802-0x13de46b5f8b000f Retrieved 156 byte(s) of data from znode /hbase/region-in-transition/1ce6d62427641e2a0e135d3d27b54b5d; data=region=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,h,1365336962398.1ce6d62427641e2a0e135d3d27b54b5d., state=RS_ZK_REGION_OPENING, servername=asf001.sp2.ygridcore.net,43802,1365336889418, createTime=1365336963086, payload.length=0 2013-04-07 12:16:03,163 DEBUG [RS_OPEN_REGION-asf001.sp2.ygridcore.net,60884,1365336878389-1] handler.OpenRegionHandler(168): Opened testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,f,1365336962398.eff8376859f5e6ba3201ec0ec9536d0f. on server:asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:16:03,163 DEBUG [RS_OPEN_REGION-asf001.sp2.ygridcore.net,60884,1365336878389-1] zookeeper.ZKAssign(721): regionserver:60884-0x13de46b5f8b0001 Attempting to transition node 11c8123e8d563f264d947ffa106e4c7c from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING 2013-04-07 12:16:03,170 DEBUG [RS_OPEN_REGION-asf001.sp2.ygridcore.net,43802,1365336889418-2] zookeeper.ZKAssign(784): regionserver:43802-0x13de46b5f8b000f Successfully transitioned node 1ce6d62427641e2a0e135d3d27b54b5d from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENED 2013-04-07 12:16:03,170 DEBUG [RS_OPEN_REGION-asf001.sp2.ygridcore.net,43802,1365336889418-2] handler.OpenRegionHandler(361): region transitioned to opened in zookeeper: {NAME => 'testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,h,1365336962398.1ce6d62427641e2a0e135d3d27b54b5d.', STARTKEY => 'h', ENDKEY => 'i', ENCODED => 1ce6d62427641e2a0e135d3d27b54b5d,}, server: asf001.sp2.ygridcore.net,43802,1365336889418 2013-04-07 12:16:03,170 DEBUG [RS_OPEN_REGION-asf001.sp2.ygridcore.net,43802,1365336889418-2] handler.OpenRegionHandler(168): Opened testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,h,1365336962398.1ce6d62427641e2a0e135d3d27b54b5d. on server:asf001.sp2.ygridcore.net,43802,1365336889418 2013-04-07 12:16:03,170 DEBUG [MASTER_OPEN_REGION-asf001.sp2.ygridcore.net,33796,1365336878036-2] zookeeper.ZKUtil(1680): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Retrieved 155 byte(s) of data from znode /hbase/region-in-transition/d4e129e51e67d798e17083623fdde133; data=region=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,,1365336962397.d4e129e51e67d798e17083623fdde133., state=RS_ZK_REGION_OPENED, servername=asf001.sp2.ygridcore.net,43802,1365336889418, createTime=1365336963136, payload.length=0 2013-04-07 12:16:03,170 DEBUG [RS_OPEN_REGION-asf001.sp2.ygridcore.net,60884,1365336878389-1] zookeeper.ZKUtil(1680): regionserver:60884-0x13de46b5f8b0001 Retrieved 156 byte(s) of data from znode /hbase/region-in-transition/11c8123e8d563f264d947ffa106e4c7c; data=region=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,b,1365336962398.11c8123e8d563f264d947ffa106e4c7c., state=M_ZK_REGION_OFFLINE, servername=asf001.sp2.ygridcore.net,60884,1365336878389, createTime=1365336962996, payload.length=0 2013-04-07 12:16:03,171 DEBUG [RS_OPEN_REGION-asf001.sp2.ygridcore.net,60884,1365336878389-2] zookeeper.ZKUtil(1680): regionserver:60884-0x13de46b5f8b0001 Retrieved 156 byte(s) of data from znode /hbase/region-in-transition/385e5846630b0cb4493758961a2c2457; data=region=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,i,1365336962398.385e5846630b0cb4493758961a2c2457., state=RS_ZK_REGION_OPENING, servername=asf001.sp2.ygridcore.net,60884,1365336878389, createTime=1365336963132, payload.length=0 2013-04-07 12:16:03,442 DEBUG [RegionServer:2;asf001.sp2.ygridcore.net,43802,1365336889418-EventThread] zookeeper.ZooKeeperWatcher(274): regionserver:43802-0x13de46b5f8b000f Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2013-04-07 12:16:03,447 DEBUG [RegionServer:2;asf001.sp2.ygridcore.net,43802,1365336889418-EventThread] zookeeper.ZKUtil(424): regionserver:43802-0x13de46b5f8b000f /hbase/master does not exist. Watcher is set. 2013-04-07 12:16:03,451 DEBUG [RegionServer:2;asf001.sp2.ygridcore.net,43802,1365336889418-EventThread] zookeeper.ZooKeeperWatcher(274): regionserver:43802-0x13de46b5f8b000f Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2013-04-07 12:16:03,454 DEBUG [RegionServer:2;asf001.sp2.ygridcore.net,43802,1365336889418-EventThread] zookeeper.ZKUtil(1680): regionserver:43802-0x13de46b5f8b000f Retrieved 43 byte(s) of data from znode /hbase/master and set watcher; PBUF\x0A%\x0A\x18asf001.sp2.y... 2013-04-07 12:16:03,585 DEBUG [RegionServer:2;asf001.sp2.ygridcore.net,43802,1365336889418-EventThread] zookeeper.ZooKeeperWatcher(274): regionserver:43802-0x13de46b5f8b000f Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2013-04-07 12:16:03,586 DEBUG [RegionServer:0;asf001.sp2.ygridcore.net,60884,1365336878389-EventThread] zookeeper.ZKUtil(424): regionserver:60884-0x13de46b5f8b0001 /hbase/running does not exist. Watcher is set. 2013-04-07 12:16:03,586 INFO [pool-1-thread-1] regionserver.HRegionServer(1545): STOPPED: Shutdown requested 2013-04-07 12:16:03,586 INFO [pool-1-thread-1] regionserver.HRegionServer(1545): STOPPED: Shutdown requested 2013-04-07 12:16:03,588 DEBUG [RegionServer:2;asf001.sp2.ygridcore.net,43802,1365336889418-EventThread] zookeeper.ZKUtil(424): regionserver:43802-0x13de46b5f8b000f /hbase/running does not exist. Watcher is set. 2013-04-07 12:16:03,588 INFO [pool-1-thread-1] regionserver.HRegionServer(1545): STOPPED: Shutdown requested 2013-04-07 12:16:03,590 INFO [RegionServer:2;asf001.sp2.ygridcore.net,43802,1365336889418] regionserver.SplitLogWorker(519): Sending interrupt to stop the worker thread 2013-04-07 12:16:03,588 INFO [SplitLogWorker-asf001.sp2.ygridcore.net,60884,1365336878389] regionserver.SplitLogWorker(205): SplitLogWorker interrupted while waiting for task, exiting: java.lang.InterruptedException 2013-04-07 12:16:03,591 INFO [RegionServer:2;asf001.sp2.ygridcore.net,43802,1365336889418] snapshot.RegionServerSnapshotManager(151): Stopping RegionServerSnapshotManager gracefully. 2013-04-07 12:16:03,591 INFO [RegionServer:2;asf001.sp2.ygridcore.net,43802,1365336889418.logRoller] regionserver.LogRoller(119): LogRoller exiting. 2013-04-07 12:16:03,591 INFO [SplitLogWorker-asf001.sp2.ygridcore.net,43802,1365336889418] regionserver.SplitLogWorker(205): SplitLogWorker interrupted while waiting for task, exiting: java.lang.InterruptedException 2013-04-07 12:16:03,591 INFO [SplitLogWorker-asf001.sp2.ygridcore.net,43802,1365336889418] regionserver.SplitLogWorker(169): SplitLogWorker asf001.sp2.ygridcore.net,43802,1365336889418 exiting 2013-04-07 12:16:03,591 INFO [RegionServer:2;asf001.sp2.ygridcore.net,43802,1365336889418.compactionChecker] hbase.Chore(94): RegionServer:2;asf001.sp2.ygridcore.net,43802,1365336889418.compactionChecker exiting 2013-04-07 12:16:03,591 DEBUG [RS_CLOSE_REGION-asf001.sp2.ygridcore.net,60884,1365336878389-1] handler.CloseRegionHandler(125): Processing close of testLogSplittingAfterMasterRecoveryDueToZKExpiry,5,1365336889784.5403a557a6168e5f3c5141eb494f6088. 2013-04-07 12:16:03,592 DEBUG [RS_CLOSE_REGION-asf001.sp2.ygridcore.net,60884,1365336878389-2] handler.CloseRegionHandler(125): Processing close of testLogSplittingAfterMasterRecoveryDueToZKExpiry,3,1365336889784.4e9a3d63237d29e6bf55bf2360834cbe. 2013-04-07 12:16:03,591 INFO [Thread-224] regionserver.MemStoreFlusher$FlushHandler(267): Thread-224 exiting 2013-04-07 12:16:03,592 DEBUG [RS_CLOSE_REGION-asf001.sp2.ygridcore.net,43802,1365336889418-1] handler.CloseRegionHandler(125): Processing close of testLogSplittingAfterMasterRecoveryDueToZKExpiry,1,1365336889784.43207168ac66dce85cf2a4f916007450. 2013-04-07 12:16:03,591 INFO [SplitLogWorker-asf001.sp2.ygridcore.net,60884,1365336878389] regionserver.SplitLogWorker(169): SplitLogWorker asf001.sp2.ygridcore.net,60884,1365336878389 exiting 2013-04-07 12:16:03,593 DEBUG [RS_CLOSE_REGION-asf001.sp2.ygridcore.net,43802,1365336889418-2] handler.CloseRegionHandler(125): Processing close of testLogSplittingAfterMasterRecoveryDueToZKExpiry,4,1365336889784.567d8ef20f90f3c1b8a889d22a1c3948. 2013-04-07 12:16:03,593 INFO [RegionServer:2;asf001.sp2.ygridcore.net,43802,1365336889418] regionserver.HRegionServer(833): stopping server asf001.sp2.ygridcore.net,43802,1365336889418 2013-04-07 12:16:03,593 DEBUG [RegionServer:2;asf001.sp2.ygridcore.net,43802,1365336889418] catalog.CatalogTracker(208): Stopping catalog tracker org.apache.hadoop.hbase.catalog.CatalogTracker@d8431c 2013-04-07 12:16:03,592 DEBUG [RS_CLOSE_REGION-asf001.sp2.ygridcore.net,43802,1365336889418-0] handler.CloseRegionHandler(125): Processing close of testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,h,1365336962398.1ce6d62427641e2a0e135d3d27b54b5d. 2013-04-07 12:16:03,593 INFO [RegionServer:2;asf001.sp2.ygridcore.net,43802,1365336889418] client.HConnectionManager$HConnectionImplementation(1672): Closing zookeeper sessionid=0x13de46b5f8b0010 2013-04-07 12:16:03,592 INFO [RegionServer:0;asf001.sp2.ygridcore.net,60884,1365336878389] regionserver.HRegionServer(833): stopping server asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:16:03,594 DEBUG [RS_CLOSE_REGION-asf001.sp2.ygridcore.net,60884,1365336878389-2] regionserver.HRegion(889): Closing testLogSplittingAfterMasterRecoveryDueToZKExpiry,3,1365336889784.4e9a3d63237d29e6bf55bf2360834cbe.: disabling compactions & flushes 2013-04-07 12:16:03,592 DEBUG [RS_CLOSE_REGION-asf001.sp2.ygridcore.net,60884,1365336878389-1] regionserver.HRegion(889): Closing testLogSplittingAfterMasterRecoveryDueToZKExpiry,5,1365336889784.5403a557a6168e5f3c5141eb494f6088.: disabling compactions & flushes 2013-04-07 12:16:03,594 DEBUG [RS_CLOSE_REGION-asf001.sp2.ygridcore.net,60884,1365336878389-2] regionserver.HRegion(911): Updates disabled for region testLogSplittingAfterMasterRecoveryDueToZKExpiry,3,1365336889784.4e9a3d63237d29e6bf55bf2360834cbe. 2013-04-07 12:16:03,594 DEBUG [RS_CLOSE_REGION-asf001.sp2.ygridcore.net,60884,1365336878389-1] regionserver.HRegion(911): Updates disabled for region testLogSplittingAfterMasterRecoveryDueToZKExpiry,5,1365336889784.5403a557a6168e5f3c5141eb494f6088. 2013-04-07 12:16:03,594 DEBUG [RS_CLOSE_REGION-asf001.sp2.ygridcore.net,43802,1365336889418-0] regionserver.HRegion(889): Closing testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,h,1365336962398.1ce6d62427641e2a0e135d3d27b54b5d.: disabling compactions & flushes 2013-04-07 12:16:03,596 DEBUG [RS_CLOSE_REGION-asf001.sp2.ygridcore.net,43802,1365336889418-0] regionserver.HRegion(911): Updates disabled for region testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,h,1365336962398.1ce6d62427641e2a0e135d3d27b54b5d. 2013-04-07 12:16:03,597 INFO [RS_CLOSE_REGION-asf001.sp2.ygridcore.net,43802,1365336889418-0] regionserver.HRegion(968): Closed testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,h,1365336962398.1ce6d62427641e2a0e135d3d27b54b5d. 2013-04-07 12:16:03,597 DEBUG [RS_CLOSE_REGION-asf001.sp2.ygridcore.net,43802,1365336889418-0] handler.CloseRegionHandler(177): Closed region testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,h,1365336962398.1ce6d62427641e2a0e135d3d27b54b5d. 2013-04-07 12:16:03,594 DEBUG [RegionServer:0;asf001.sp2.ygridcore.net,60884,1365336878389] catalog.CatalogTracker(208): Stopping catalog tracker org.apache.hadoop.hbase.catalog.CatalogTracker@17535b6 2013-04-07 12:16:03,594 DEBUG [RS_CLOSE_REGION-asf001.sp2.ygridcore.net,43802,1365336889418-2] regionserver.HRegion(889): Closing testLogSplittingAfterMasterRecoveryDueToZKExpiry,4,1365336889784.567d8ef20f90f3c1b8a889d22a1c3948.: disabling compactions & flushes 2013-04-07 12:16:03,594 DEBUG [RS_CLOSE_REGION-asf001.sp2.ygridcore.net,43802,1365336889418-1] regionserver.HRegion(889): Closing testLogSplittingAfterMasterRecoveryDueToZKExpiry,1,1365336889784.43207168ac66dce85cf2a4f916007450.: disabling compactions & flushes 2013-04-07 12:16:03,597 DEBUG [RS_CLOSE_REGION-asf001.sp2.ygridcore.net,43802,1365336889418-2] regionserver.HRegion(911): Updates disabled for region testLogSplittingAfterMasterRecoveryDueToZKExpiry,4,1365336889784.567d8ef20f90f3c1b8a889d22a1c3948. 2013-04-07 12:16:03,597 DEBUG [RS_CLOSE_REGION-asf001.sp2.ygridcore.net,43802,1365336889418-1] regionserver.HRegion(911): Updates disabled for region testLogSplittingAfterMasterRecoveryDueToZKExpiry,1,1365336889784.43207168ac66dce85cf2a4f916007450. 2013-04-07 12:16:03,597 DEBUG [RS_CLOSE_REGION-asf001.sp2.ygridcore.net,43802,1365336889418-0] handler.CloseRegionHandler(125): Processing close of testLogSplittingAfterMasterRecoveryDueToZKExpiry,2,1365336889784.55884b67b9203a71bdf8c0e3de70452d. 2013-04-07 12:16:03,596 INFO [RS_CLOSE_REGION-asf001.sp2.ygridcore.net,60884,1365336878389-1] regionserver.HRegion(968): Closed testLogSplittingAfterMasterRecoveryDueToZKExpiry,5,1365336889784.5403a557a6168e5f3c5141eb494f6088. 2013-04-07 12:16:03,598 DEBUG [RS_CLOSE_REGION-asf001.sp2.ygridcore.net,43802,1365336889418-0] regionserver.HRegion(889): Closing testLogSplittingAfterMasterRecoveryDueToZKExpiry,2,1365336889784.55884b67b9203a71bdf8c0e3de70452d.: disabling compactions & flushes 2013-04-07 12:16:03,598 DEBUG [RS_CLOSE_REGION-asf001.sp2.ygridcore.net,60884,1365336878389-1] handler.CloseRegionHandler(177): Closed region testLogSplittingAfterMasterRecoveryDueToZKExpiry,5,1365336889784.5403a557a6168e5f3c5141eb494f6088. 2013-04-07 12:16:03,598 DEBUG [RS_CLOSE_REGION-asf001.sp2.ygridcore.net,43802,1365336889418-0] regionserver.HRegion(911): Updates disabled for region testLogSplittingAfterMasterRecoveryDueToZKExpiry,2,1365336889784.55884b67b9203a71bdf8c0e3de70452d. 2013-04-07 12:16:03,598 DEBUG [RS_CLOSE_REGION-asf001.sp2.ygridcore.net,60884,1365336878389-1] handler.CloseRegionHandler(125): Processing close of testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,b,1365336962398.11c8123e8d563f264d947ffa106e4c7c. 2013-04-07 12:16:03,596 INFO [RS_CLOSE_REGION-asf001.sp2.ygridcore.net,60884,1365336878389-2] regionserver.HRegion(968): Closed testLogSplittingAfterMasterRecoveryDueToZKExpiry,3,1365336889784.4e9a3d63237d29e6bf55bf2360834cbe. 2013-04-07 12:16:03,598 DEBUG [RS_CLOSE_REGION-asf001.sp2.ygridcore.net,60884,1365336878389-1] regionserver.HRegion(889): Closing testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,b,1365336962398.11c8123e8d563f264d947ffa106e4c7c.: disabling compactions & flushes 2013-04-07 12:16:03,599 DEBUG [RS_CLOSE_REGION-asf001.sp2.ygridcore.net,60884,1365336878389-1] regionserver.HRegion(911): Updates disabled for region testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,b,1365336962398.11c8123e8d563f264d947ffa106e4c7c. 2013-04-07 12:16:03,599 INFO [RS_CLOSE_REGION-asf001.sp2.ygridcore.net,60884,1365336878389-1] regionserver.HRegion(968): Closed testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,b,1365336962398.11c8123e8d563f264d947ffa106e4c7c. 2013-04-07 12:16:03,599 DEBUG [RS_CLOSE_REGION-asf001.sp2.ygridcore.net,60884,1365336878389-1] handler.CloseRegionHandler(177): Closed region testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,b,1365336962398.11c8123e8d563f264d947ffa106e4c7c. 2013-04-07 12:16:03,599 DEBUG [RS_CLOSE_REGION-asf001.sp2.ygridcore.net,60884,1365336878389-1] handler.CloseRegionHandler(125): Processing close of testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,i,1365336962398.385e5846630b0cb4493758961a2c2457. 2013-04-07 12:16:03,598 INFO [StoreCloserThread-testLogSplittingAfterMasterRecoveryDueToZKExpiry,2,1365336889784.55884b67b9203a71bdf8c0e3de70452d.-1] regionserver.HStore(622): Closed col 2013-04-07 12:16:03,598 DEBUG [RS_CLOSE_REGION-asf001.sp2.ygridcore.net,60884,1365336878389-2] handler.CloseRegionHandler(177): Closed region testLogSplittingAfterMasterRecoveryDueToZKExpiry,3,1365336889784.4e9a3d63237d29e6bf55bf2360834cbe. 2013-04-07 12:16:03,601 INFO [RS_CLOSE_REGION-asf001.sp2.ygridcore.net,43802,1365336889418-0] regionserver.HRegion(968): Closed testLogSplittingAfterMasterRecoveryDueToZKExpiry,2,1365336889784.55884b67b9203a71bdf8c0e3de70452d. 2013-04-07 12:16:03,601 DEBUG [RS_CLOSE_REGION-asf001.sp2.ygridcore.net,60884,1365336878389-2] handler.CloseRegionHandler(125): Processing close of test1365336888306,,1365336888309.da419688e5c79938e207c0d3c056a9ed. 2013-04-07 12:16:03,602 DEBUG [RS_CLOSE_REGION-asf001.sp2.ygridcore.net,43802,1365336889418-0] handler.CloseRegionHandler(177): Closed region testLogSplittingAfterMasterRecoveryDueToZKExpiry,2,1365336889784.55884b67b9203a71bdf8c0e3de70452d. 2013-04-07 12:16:03,602 DEBUG [RS_CLOSE_REGION-asf001.sp2.ygridcore.net,60884,1365336878389-2] regionserver.HRegion(889): Closing test1365336888306,,1365336888309.da419688e5c79938e207c0d3c056a9ed.: disabling compactions & flushes 2013-04-07 12:16:03,602 DEBUG [RS_CLOSE_REGION-asf001.sp2.ygridcore.net,43802,1365336889418-0] handler.CloseRegionHandler(125): Processing close of testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,c,1365336962398.4432e800611c00a4ae168862a273fc70. 2013-04-07 12:16:03,602 DEBUG [RS_CLOSE_REGION-asf001.sp2.ygridcore.net,60884,1365336878389-2] regionserver.HRegion(911): Updates disabled for region test1365336888306,,1365336888309.da419688e5c79938e207c0d3c056a9ed. 2013-04-07 12:16:03,602 DEBUG [RS_CLOSE_REGION-asf001.sp2.ygridcore.net,43802,1365336889418-0] regionserver.HRegion(889): Closing testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,c,1365336962398.4432e800611c00a4ae168862a273fc70.: disabling compactions & flushes 2013-04-07 12:16:03,602 DEBUG [RS_CLOSE_REGION-asf001.sp2.ygridcore.net,60884,1365336878389-2] regionserver.HRegion(1391): Started memstore flush for test1365336888306,,1365336888309.da419688e5c79938e207c0d3c056a9ed., current region memstore size 136.0 2013-04-07 12:16:03,603 DEBUG [RS_CLOSE_REGION-asf001.sp2.ygridcore.net,43802,1365336889418-0] regionserver.HRegion(911): Updates disabled for region testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,c,1365336962398.4432e800611c00a4ae168862a273fc70. 2013-04-07 12:16:03,598 INFO [StoreCloserThread-testLogSplittingAfterMasterRecoveryDueToZKExpiry,1,1365336889784.43207168ac66dce85cf2a4f916007450.-1] regionserver.HStore(622): Closed col 2013-04-07 12:16:03,597 INFO [StoreCloserThread-testLogSplittingAfterMasterRecoveryDueToZKExpiry,4,1365336889784.567d8ef20f90f3c1b8a889d22a1c3948.-1] regionserver.HStore(622): Closed col 2013-04-07 12:16:03,603 INFO [RS_CLOSE_REGION-asf001.sp2.ygridcore.net,43802,1365336889418-1] regionserver.HRegion(968): Closed testLogSplittingAfterMasterRecoveryDueToZKExpiry,1,1365336889784.43207168ac66dce85cf2a4f916007450. 2013-04-07 12:16:03,603 INFO [RS_CLOSE_REGION-asf001.sp2.ygridcore.net,43802,1365336889418-0] regionserver.HRegion(968): Closed testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,c,1365336962398.4432e800611c00a4ae168862a273fc70. 2013-04-07 12:16:03,603 DEBUG [RS_CLOSE_REGION-asf001.sp2.ygridcore.net,43802,1365336889418-1] handler.CloseRegionHandler(177): Closed region testLogSplittingAfterMasterRecoveryDueToZKExpiry,1,1365336889784.43207168ac66dce85cf2a4f916007450. 2013-04-07 12:16:03,603 DEBUG [RS_CLOSE_REGION-asf001.sp2.ygridcore.net,43802,1365336889418-0] handler.CloseRegionHandler(177): Closed region testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,c,1365336962398.4432e800611c00a4ae168862a273fc70. 2013-04-07 12:16:03,604 DEBUG [RS_CLOSE_REGION-asf001.sp2.ygridcore.net,43802,1365336889418-1] handler.CloseRegionHandler(125): Processing close of testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,,1365336962397.d4e129e51e67d798e17083623fdde133. 2013-04-07 12:16:03,603 DEBUG [RS_CLOSE_REGION-asf001.sp2.ygridcore.net,60884,1365336878389-2] regionserver.HRegion(1447): Finished snapshotting test1365336888306,,1365336888309.da419688e5c79938e207c0d3c056a9ed., commencing wait for mvcc, flushsize=136 2013-04-07 12:16:03,605 DEBUG [RS_CLOSE_REGION-asf001.sp2.ygridcore.net,43802,1365336889418-1] regionserver.HRegion(889): Closing testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,,1365336962397.d4e129e51e67d798e17083623fdde133.: disabling compactions & flushes 2013-04-07 12:16:03,602 DEBUG [RS_CLOSE_REGION-asf001.sp2.ygridcore.net,60884,1365336878389-0] regionserver.HRegion(1447): Finished snapshotting test1365336881340,,1365336881424.c9e9e308a02801bb891879b6cf471381., commencing wait for mvcc, flushsize=136 2013-04-07 12:16:03,601 DEBUG [RS_CLOSE_REGION-asf001.sp2.ygridcore.net,60884,1365336878389-1] regionserver.HRegion(911): Updates disabled for region testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,i,1365336962398.385e5846630b0cb4493758961a2c2457. 2013-04-07 12:16:03,605 DEBUG [RS_CLOSE_REGION-asf001.sp2.ygridcore.net,60884,1365336878389-0] regionserver.HRegion(1463): Finished snapshotting, commencing flushing stores 2013-04-07 12:16:03,605 DEBUG [RS_CLOSE_REGION-asf001.sp2.ygridcore.net,43802,1365336889418-1] regionserver.HRegion(911): Updates disabled for region testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,,1365336962397.d4e129e51e67d798e17083623fdde133. 2013-04-07 12:16:03,605 DEBUG [RS_CLOSE_REGION-asf001.sp2.ygridcore.net,60884,1365336878389-2] regionserver.HRegion(1463): Finished snapshotting, commencing flushing stores 2013-04-07 12:16:03,603 INFO [RS_CLOSE_REGION-asf001.sp2.ygridcore.net,43802,1365336889418-2] regionserver.HRegion(968): Closed testLogSplittingAfterMasterRecoveryDueToZKExpiry,4,1365336889784.567d8ef20f90f3c1b8a889d22a1c3948. 2013-04-07 12:16:03,606 INFO [RS_CLOSE_REGION-asf001.sp2.ygridcore.net,43802,1365336889418-1] regionserver.HRegion(968): Closed testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,,1365336962397.d4e129e51e67d798e17083623fdde133. 2013-04-07 12:16:03,606 DEBUG [RS_CLOSE_REGION-asf001.sp2.ygridcore.net,43802,1365336889418-2] handler.CloseRegionHandler(177): Closed region testLogSplittingAfterMasterRecoveryDueToZKExpiry,4,1365336889784.567d8ef20f90f3c1b8a889d22a1c3948. 2013-04-07 12:16:03,606 DEBUG [RS_CLOSE_REGION-asf001.sp2.ygridcore.net,43802,1365336889418-1] handler.CloseRegionHandler(177): Closed region testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,,1365336962397.d4e129e51e67d798e17083623fdde133. 2013-04-07 12:16:03,605 INFO [RS_CLOSE_REGION-asf001.sp2.ygridcore.net,60884,1365336878389-1] regionserver.HRegion(968): Closed testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,i,1365336962398.385e5846630b0cb4493758961a2c2457. 2013-04-07 12:16:03,606 DEBUG [RS_CLOSE_REGION-asf001.sp2.ygridcore.net,60884,1365336878389-1] handler.CloseRegionHandler(177): Closed region testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,i,1365336962398.385e5846630b0cb4493758961a2c2457. 2013-04-07 12:16:03,613 DEBUG [RS_CLOSE_REGION-asf001.sp2.ygridcore.net,60884,1365336878389-0] util.FSUtils(211): Creating file=hdfs://localhost:60405/user/jenkins/hbase/test1365336881340/c9e9e308a02801bb891879b6cf471381/.tmp/1f3905c6a4f84521a315a19c4afdc124 with permission=rwxrwxrwx 2013-04-07 12:16:03,613 DEBUG [RS_CLOSE_REGION-asf001.sp2.ygridcore.net,60884,1365336878389-2] util.FSUtils(211): Creating file=hdfs://localhost:60405/user/jenkins/hbase/test1365336888306/da419688e5c79938e207c0d3c056a9ed/.tmp/4aa4e17c77c24a9aa3b49da119621110 with permission=rwxrwxrwx 2013-04-07 12:16:03,624 DEBUG [RS_CLOSE_REGION-asf001.sp2.ygridcore.net,60884,1365336878389-0] hfile.HFileWriterV2(146): Initialized with CacheConfig:enabled [cacheDataOnRead=true] [cacheDataOnWrite=false] [cacheIndexesOnWrite=false] [cacheBloomsOnWrite=false] [cacheEvictOnClose=false] [cacheCompressed=false] 2013-04-07 12:16:03,624 DEBUG [RS_CLOSE_REGION-asf001.sp2.ygridcore.net,60884,1365336878389-2] hfile.HFileWriterV2(146): Initialized with CacheConfig:enabled [cacheDataOnRead=true] [cacheDataOnWrite=false] [cacheIndexesOnWrite=false] [cacheBloomsOnWrite=false] [cacheEvictOnClose=false] [cacheCompressed=false] 2013-04-07 12:16:03,628 INFO [RS_CLOSE_REGION-asf001.sp2.ygridcore.net,60884,1365336878389-0] regionserver.StoreFile$Writer(813): Delete Family Bloom filter type for hdfs://localhost:60405/user/jenkins/hbase/test1365336881340/c9e9e308a02801bb891879b6cf471381/.tmp/1f3905c6a4f84521a315a19c4afdc124: CompoundBloomFilterWriter 2013-04-07 12:16:03,628 INFO [RS_CLOSE_REGION-asf001.sp2.ygridcore.net,60884,1365336878389-2] regionserver.StoreFile$Writer(813): Delete Family Bloom filter type for hdfs://localhost:60405/user/jenkins/hbase/test1365336888306/da419688e5c79938e207c0d3c056a9ed/.tmp/4aa4e17c77c24a9aa3b49da119621110: CompoundBloomFilterWriter 2013-04-07 12:16:03,648 INFO [RS_CLOSE_REGION-asf001.sp2.ygridcore.net,60884,1365336878389-2] regionserver.StoreFile$Writer(1033): NO General Bloom and NO DeleteFamily was added to HFile (hdfs://localhost:60405/user/jenkins/hbase/test1365336888306/da419688e5c79938e207c0d3c056a9ed/.tmp/4aa4e17c77c24a9aa3b49da119621110) 2013-04-07 12:16:03,649 INFO [RS_CLOSE_REGION-asf001.sp2.ygridcore.net,60884,1365336878389-2] regionserver.HStore(791): Flushed , sequenceid=5, memsize=136.0, into tmp file hdfs://localhost:60405/user/jenkins/hbase/test1365336888306/da419688e5c79938e207c0d3c056a9ed/.tmp/4aa4e17c77c24a9aa3b49da119621110 2013-04-07 12:16:03,650 INFO [RS_CLOSE_REGION-asf001.sp2.ygridcore.net,60884,1365336878389-0] regionserver.StoreFile$Writer(1033): NO General Bloom and NO DeleteFamily was added to HFile (hdfs://localhost:60405/user/jenkins/hbase/test1365336881340/c9e9e308a02801bb891879b6cf471381/.tmp/1f3905c6a4f84521a315a19c4afdc124) 2013-04-07 12:16:03,650 INFO [RS_CLOSE_REGION-asf001.sp2.ygridcore.net,60884,1365336878389-0] regionserver.HStore(791): Flushed , sequenceid=4, memsize=136.0, into tmp file hdfs://localhost:60405/user/jenkins/hbase/test1365336881340/c9e9e308a02801bb891879b6cf471381/.tmp/1f3905c6a4f84521a315a19c4afdc124 2013-04-07 12:16:03,675 DEBUG [RS_CLOSE_REGION-asf001.sp2.ygridcore.net,60884,1365336878389-0] regionserver.HRegionFileSystem(338): Committing store file hdfs://localhost:60405/user/jenkins/hbase/test1365336881340/c9e9e308a02801bb891879b6cf471381/.tmp/1f3905c6a4f84521a315a19c4afdc124 as hdfs://localhost:60405/user/jenkins/hbase/test1365336881340/c9e9e308a02801bb891879b6cf471381/fam/1f3905c6a4f84521a315a19c4afdc124 2013-04-07 12:16:03,676 DEBUG [RS_CLOSE_REGION-asf001.sp2.ygridcore.net,60884,1365336878389-2] regionserver.HRegionFileSystem(338): Committing store file hdfs://localhost:60405/user/jenkins/hbase/test1365336888306/da419688e5c79938e207c0d3c056a9ed/.tmp/4aa4e17c77c24a9aa3b49da119621110 as hdfs://localhost:60405/user/jenkins/hbase/test1365336888306/da419688e5c79938e207c0d3c056a9ed/fam/4aa4e17c77c24a9aa3b49da119621110 2013-04-07 12:16:03,686 INFO [RS_CLOSE_REGION-asf001.sp2.ygridcore.net,60884,1365336878389-0] regionserver.HStore(823): Added hdfs://localhost:60405/user/jenkins/hbase/test1365336881340/c9e9e308a02801bb891879b6cf471381/fam/1f3905c6a4f84521a315a19c4afdc124, entries=1, sequenceid=4, filesize=799.0 2013-04-07 12:16:03,687 INFO [RS_CLOSE_REGION-asf001.sp2.ygridcore.net,60884,1365336878389-0] regionserver.HRegion(1534): Finished memstore flush of ~136.0/136, currentsize=0.0/0 for region test1365336881340,,1365336881424.c9e9e308a02801bb891879b6cf471381. in 85ms, sequenceid=4, compaction requested=false 2013-04-07 12:16:03,687 INFO [RS_CLOSE_REGION-asf001.sp2.ygridcore.net,60884,1365336878389-2] regionserver.HStore(823): Added hdfs://localhost:60405/user/jenkins/hbase/test1365336888306/da419688e5c79938e207c0d3c056a9ed/fam/4aa4e17c77c24a9aa3b49da119621110, entries=1, sequenceid=5, filesize=799.0 2013-04-07 12:16:03,689 INFO [RS_CLOSE_REGION-asf001.sp2.ygridcore.net,60884,1365336878389-2] regionserver.HRegion(1534): Finished memstore flush of ~136.0/136, currentsize=0.0/0 for region test1365336888306,,1365336888309.da419688e5c79938e207c0d3c056a9ed. in 87ms, sequenceid=5, compaction requested=false 2013-04-07 12:16:03,689 INFO [StoreCloserThread-test1365336881340,,1365336881424.c9e9e308a02801bb891879b6cf471381.-1] regionserver.HStore(622): Closed fam 2013-04-07 12:16:03,689 INFO [RS_CLOSE_REGION-asf001.sp2.ygridcore.net,60884,1365336878389-0] regionserver.HRegion(968): Closed test1365336881340,,1365336881424.c9e9e308a02801bb891879b6cf471381. 2013-04-07 12:16:03,689 DEBUG [RS_CLOSE_REGION-asf001.sp2.ygridcore.net,60884,1365336878389-0] handler.CloseRegionHandler(177): Closed region test1365336881340,,1365336881424.c9e9e308a02801bb891879b6cf471381. 2013-04-07 12:16:03,690 INFO [StoreCloserThread-test1365336888306,,1365336888309.da419688e5c79938e207c0d3c056a9ed.-1] regionserver.HStore(622): Closed fam 2013-04-07 12:16:03,690 INFO [RS_CLOSE_REGION-asf001.sp2.ygridcore.net,60884,1365336878389-2] regionserver.HRegion(968): Closed test1365336888306,,1365336888309.da419688e5c79938e207c0d3c056a9ed. 2013-04-07 12:16:03,690 DEBUG [RS_CLOSE_REGION-asf001.sp2.ygridcore.net,60884,1365336878389-2] handler.CloseRegionHandler(177): Closed region test1365336888306,,1365336888309.da419688e5c79938e207c0d3c056a9ed. 2013-04-07 12:16:03,698 INFO [RegionServer:2;asf001.sp2.ygridcore.net,43802,1365336889418] snapshot.RegionServerSnapshotManager(151): Stopping RegionServerSnapshotManager gracefully. 2013-04-07 12:16:03,698 INFO [RegionServer:2;asf001.sp2.ygridcore.net,43802,1365336889418] regionserver.HRegionServer(859): stopping server asf001.sp2.ygridcore.net,43802,1365336889418; all regions closed. 2013-04-07 12:16:03,699 INFO [RegionServer:2;asf001.sp2.ygridcore.net,43802,1365336889418.logSyncer] wal.FSHLog$LogSyncer(1030): RegionServer:2;asf001.sp2.ygridcore.net,43802,1365336889418.logSyncer exiting 2013-04-07 12:16:03,699 DEBUG [RegionServer:2;asf001.sp2.ygridcore.net,43802,1365336889418] wal.FSHLog(836): closing hlog writer in hdfs://localhost:60405/user/jenkins/hbase/.logs/asf001.sp2.ygridcore.net,43802,1365336889418 2013-04-07 12:16:03,724 DEBUG [RegionServer:2;asf001.sp2.ygridcore.net,43802,1365336889418] wal.FSHLog(796): Moved 1 log files to /user/jenkins/hbase/.oldlogs 2013-04-07 12:16:03,783 INFO [RegionServer:2;asf001.sp2.ygridcore.net,43802,1365336889418.leaseChecker] regionserver.Leases(124): RegionServer:2;asf001.sp2.ygridcore.net,43802,1365336889418.leaseChecker closing leases 2013-04-07 12:16:03,783 INFO [RegionServer:2;asf001.sp2.ygridcore.net,43802,1365336889418.leaseChecker] regionserver.Leases(131): RegionServer:2;asf001.sp2.ygridcore.net,43802,1365336889418.leaseChecker closed leases 2013-04-07 12:16:03,827 INFO [RegionServer:2;asf001.sp2.ygridcore.net,43802,1365336889418] regionserver.Leases(124): RegionServer:2;asf001.sp2.ygridcore.net,43802,1365336889418 closing leases 2013-04-07 12:16:03,827 INFO [RegionServer:2;asf001.sp2.ygridcore.net,43802,1365336889418] regionserver.Leases(131): RegionServer:2;asf001.sp2.ygridcore.net,43802,1365336889418 closed leases 2013-04-07 12:16:03,827 INFO [RegionServer:2;asf001.sp2.ygridcore.net,43802,1365336889418] regionserver.CompactSplitThread(318): Waiting for Split Thread to finish... 2013-04-07 12:16:03,827 INFO [RegionServer:2;asf001.sp2.ygridcore.net,43802,1365336889418] regionserver.CompactSplitThread(318): Waiting for Merge Thread to finish... 2013-04-07 12:16:03,827 INFO [RegionServer:2;asf001.sp2.ygridcore.net,43802,1365336889418] regionserver.CompactSplitThread(318): Waiting for Large Compaction Thread to finish... 2013-04-07 12:16:03,827 INFO [RegionServer:2;asf001.sp2.ygridcore.net,43802,1365336889418] regionserver.CompactSplitThread(318): Waiting for Small Compaction Thread to finish... 2013-04-07 12:16:03,833 INFO [RegionServer:2;asf001.sp2.ygridcore.net,43802,1365336889418] regionserver.HRegionServer(888): stopping server asf001.sp2.ygridcore.net,43802,1365336889418; zookeeper connection closed. 2013-04-07 12:16:03,833 INFO [RegionServer:2;asf001.sp2.ygridcore.net,43802,1365336889418] regionserver.HRegionServer(891): RegionServer:2;asf001.sp2.ygridcore.net,43802,1365336889418 exiting 2013-04-07 12:16:03,833 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@8d5485] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(186): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@8d5485 2013-04-07 12:16:05,148 DEBUG [Thread-214-EventThread] zookeeper.ZooKeeperWatcher(274): unittest-0x13de46b5f8b0011 Received ZooKeeper Event, type=None, state=Disconnected, path=null 2013-04-07 12:16:05,148 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(274): hconnection-0xfe04c6-0x13de46b5f8b0007 Received ZooKeeper Event, type=None, state=Disconnected, path=null 2013-04-07 12:16:05,148 DEBUG [Thread-214-EventThread] zookeeper.ZooKeeperWatcher(356): unittest-0x13de46b5f8b0011 Received Disconnected from ZooKeeper, ignoring 2013-04-07 12:16:05,148 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(356): hconnection-0xfe04c6-0x13de46b5f8b0007 Received Disconnected from ZooKeeper, ignoring 2013-04-07 12:16:02,299 INFO [pool-1-thread-1] hbase.ResourceChecker(147): before: TestZooKeeper#testRegionAssignmentAfterMasterRecoveryDueToZKExpiry Thread=226, OpenFileDescriptor=359, MaxFileDescriptor=60000, SystemLoadAverage=306, ProcessCount=175, AvailableMemoryMB=7197, ConnectionCount=5 2013-04-07 12:16:02,299 INFO [pool-1-thread-1] hbase.LocalHBaseCluster(243): Not alive RegionServer:1;asf001.sp2.ygridcore.net,35684,1365336878445 2013-04-07 12:16:02,319 DEBUG [pool-1-thread-1] client.HConnectionManager(2673): HConnection retries=100 2013-04-07 12:16:02,325 DEBUG [RegionServer:3;asf001.sp2.ygridcore.net,49518,1365336962323] zookeeper.ZKConfig(84): Skipped reading ZK properties file 'zoo.cfg' since 'hbase.config.read.zookeeper.config' was not set to true 2013-04-07 12:16:02,326 DEBUG [RegionServer:3;asf001.sp2.ygridcore.net,49518,1365336962323] zookeeper.ZKUtil(119): regionserver:49518 opening connection to ZooKeeper with ensemble (localhost:51597) 2013-04-07 12:16:02,327 INFO [RegionServer:3;asf001.sp2.ygridcore.net,49518,1365336962323] zookeeper.RecoverableZooKeeper(119): The identifier of this process is regionserver:49518 2013-04-07 12:16:02,331 DEBUG [RegionServer:3;asf001.sp2.ygridcore.net,49518,1365336962323-EventThread] zookeeper.ZooKeeperWatcher(274): regionserver:49518 Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2013-04-07 12:16:02,331 DEBUG [RegionServer:3;asf001.sp2.ygridcore.net,49518,1365336962323] zookeeper.ZKUtil(422): regionserver:49518 Set watcher on existing znode /hbase/master 2013-04-07 12:16:02,332 DEBUG [RegionServer:3;asf001.sp2.ygridcore.net,49518,1365336962323] zookeeper.ZKUtil(1680): regionserver:49518 Retrieved 43 byte(s) of data from znode /hbase/master and set watcher; PBUF\x0A%\x0A\x18asf001.sp2.y... 2013-04-07 12:16:02,333 DEBUG [RegionServer:3;asf001.sp2.ygridcore.net,49518,1365336962323-EventThread] zookeeper.ZooKeeperWatcher(351): regionserver:49518-0x13de46b5f8b0012 connected 2013-04-07 12:16:02,334 DEBUG [RegionServer:3;asf001.sp2.ygridcore.net,49518,1365336962323] zookeeper.ZKUtil(422): regionserver:49518-0x13de46b5f8b0012 Set watcher on existing znode /hbase/running 2013-04-07 12:16:02,334 DEBUG [RegionServer:3;asf001.sp2.ygridcore.net,49518,1365336962323] zookeeper.ZKUtil(1680): regionserver:49518-0x13de46b5f8b0012 Retrieved 34 byte(s) of data from znode /hbase/running and set watcher; PBUF\x0A\x1CSun Apr 07 12:14:... 2013-04-07 12:16:02,335 DEBUG [RegionServer:3;asf001.sp2.ygridcore.net,49518,1365336962323] zookeeper.ZKConfig(84): Skipped reading ZK properties file 'zoo.cfg' since 'hbase.config.read.zookeeper.config' was not set to true 2013-04-07 12:16:02,336 DEBUG [RegionServer:3;asf001.sp2.ygridcore.net,49518,1365336962323] zookeeper.ZKUtil(119): hconnection-0x1f357d3 opening connection to ZooKeeper with ensemble (localhost:51597) 2013-04-07 12:16:02,337 INFO [RegionServer:3;asf001.sp2.ygridcore.net,49518,1365336962323] zookeeper.RecoverableZooKeeper(119): The identifier of this process is hconnection-0x1f357d3 2013-04-07 12:16:02,338 DEBUG [RegionServer:3;asf001.sp2.ygridcore.net,49518,1365336962323-EventThread] zookeeper.ZooKeeperWatcher(274): hconnection-0x1f357d3 Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2013-04-07 12:16:02,339 DEBUG [RegionServer:3;asf001.sp2.ygridcore.net,49518,1365336962323] zookeeper.ZKUtil(1680): hconnection-0x1f357d3 Retrieved 42 byte(s) of data from znode /hbase/hbaseid; data=PBUF\x0A$cd034c7a-08a3-4d5e-8... 2013-04-07 12:16:02,339 DEBUG [RegionServer:3;asf001.sp2.ygridcore.net,49518,1365336962323-EventThread] zookeeper.ZooKeeperWatcher(351): hconnection-0x1f357d3-0x13de46b5f8b0013 connected 2013-04-07 12:16:02,339 INFO [RegionServer:3;asf001.sp2.ygridcore.net,49518,1365336962323] client.HConnectionManager$HConnectionImplementation(699): ClusterId is cd034c7a-08a3-4d5e-8eea-f9d6aa6dd983 2013-04-07 12:16:02,340 DEBUG [RegionServer:3;asf001.sp2.ygridcore.net,49518,1365336962323] catalog.CatalogTracker(192): Starting catalog tracker org.apache.hadoop.hbase.catalog.CatalogTracker@1945696 2013-04-07 12:16:02,341 DEBUG [RegionServer:3;asf001.sp2.ygridcore.net,49518,1365336962323] zookeeper.ZKUtil(422): regionserver:49518-0x13de46b5f8b0012 Set watcher on existing znode /hbase/meta-region-server 2013-04-07 12:16:02,342 DEBUG [RegionServer:3;asf001.sp2.ygridcore.net,49518,1365336962323] zookeeper.ZKUtil(1680): regionserver:49518-0x13de46b5f8b0012 Retrieved 43 byte(s) of data from znode /hbase/meta-region-server and set watcher; asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:16:02,344 DEBUG [RegionServer:3;asf001.sp2.ygridcore.net,49518,1365336962323] zookeeper.ZKUtil(1680): regionserver:49518-0x13de46b5f8b0012 Retrieved 42 byte(s) of data from znode /hbase/hbaseid; data=PBUF\x0A$cd034c7a-08a3-4d5e-8... 2013-04-07 12:16:02,344 INFO [RegionServer:3;asf001.sp2.ygridcore.net,49518,1365336962323] regionserver.HRegionServer(634): ClusterId : cd034c7a-08a3-4d5e-8eea-f9d6aa6dd983 2013-04-07 12:16:02,346 INFO [RegionServer:3;asf001.sp2.ygridcore.net,49518,1365336962323] zookeeper.RecoverableZooKeeper(460): Node /hbase/online-snapshot/acquired already exists and this is not a retry 2013-04-07 12:16:02,347 INFO [RegionServer:3;asf001.sp2.ygridcore.net,49518,1365336962323] regionserver.MemStoreFlusher(117): globalMemStoreLimit=675.6m, globalMemStoreLimitLowMark=591.1m, maxHeap=1.6g 2013-04-07 12:16:02,348 INFO [RegionServer:3;asf001.sp2.ygridcore.net,49518,1365336962323] regionserver.HRegionServer$CompactionChecker(1237): Runs every 16mins, 40sec 2013-04-07 12:16:02,348 INFO [RegionServer:3;asf001.sp2.ygridcore.net,49518,1365336962323] regionserver.HRegionServer(1737): Attempting connect to Master server at asf001.sp2.ygridcore.net,33796,1365336878036 2013-04-07 12:16:02,348 INFO [RegionServer:3;asf001.sp2.ygridcore.net,49518,1365336962323] regionserver.HRegionServer(1744): Connected to master at asf001.sp2.ygridcore.net/67.195.138.31:33796 2013-04-07 12:16:02,348 INFO [RegionServer:3;asf001.sp2.ygridcore.net,49518,1365336962323] regionserver.HRegionServer(1790): Telling master at asf001.sp2.ygridcore.net,33796,1365336878036 that we are up with port=49518, startcode=1365336962323 2013-04-07 12:16:02,350 INFO [IPC Server handler 2 on 33796] master.ServerManager(352): Registering server=asf001.sp2.ygridcore.net,49518,1365336962323 2013-04-07 12:16:02,351 DEBUG [RegionServer:3;asf001.sp2.ygridcore.net,49518,1365336962323] regionserver.HRegionServer(1084): Config from master: hbase.rootdir=hdfs://localhost:60405/user/jenkins/hbase 2013-04-07 12:16:02,351 DEBUG [RegionServer:3;asf001.sp2.ygridcore.net,49518,1365336962323] regionserver.HRegionServer(1084): Config from master: fs.default.name=hdfs://localhost:60405 2013-04-07 12:16:02,351 INFO [RegionServer:3;asf001.sp2.ygridcore.net,49518,1365336962323] regionserver.HRegionServer(1077): Master passed us hostname to use. Was=asf001.sp2.ygridcore.net, Now=asf001.sp2.ygridcore.net 2013-04-07 12:16:02,353 WARN [RegionServer:3;asf001.sp2.ygridcore.net,49518,1365336962323] hbase.ZNodeClearer(58): No filename given to save the znode used, it won't be saved (Environment variable HBASE_ZNODE_FILE is not set). 2013-04-07 12:16:02,356 DEBUG [RegionServer:3;asf001.sp2.ygridcore.net,49518,1365336962323] fs.HFileSystem(199): Starting addLocationsOrderInterceptor with class class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2013-04-07 12:16:02,357 INFO [RegionServer:3;asf001.sp2.ygridcore.net,49518,1365336962323] fs.HFileSystem(246): Added intercepting call to namenode#getBlockLocations 2013-04-07 12:16:02,357 DEBUG [RegionServer:3;asf001.sp2.ygridcore.net,49518,1365336962323] regionserver.HRegionServer(1300): logdir=hdfs://localhost:60405/user/jenkins/hbase/.logs/asf001.sp2.ygridcore.net,49518,1365336962323 2013-04-07 12:16:02,358 INFO [RegionServer:3;asf001.sp2.ygridcore.net,49518,1365336962323] wal.FSHLog(354): HLog configuration: blocksize=64 MB, rollsize=60.8 MB, enabled=true, optionallogflushinternal=1000ms 2013-04-07 12:16:02,367 DEBUG [RegionServer:3;asf001.sp2.ygridcore.net,49518,1365336962323] wal.SequenceFileLogWriter(189): using new createWriter -- HADOOP-6840 2013-04-07 12:16:02,368 DEBUG [RegionServer:3;asf001.sp2.ygridcore.net,49518,1365336962323] wal.SequenceFileLogWriter(193): Path=hdfs://localhost:60405/user/jenkins/hbase/.logs/asf001.sp2.ygridcore.net,49518,1365336962323/asf001.sp2.ygridcore.net%2C49518%2C1365336962323.1365336962362, compression=false 2013-04-07 12:16:02,368 INFO [RegionServer:3;asf001.sp2.ygridcore.net,49518,1365336962323] wal.FSHLog(552): Rolled log; new path=/user/jenkins/hbase/.logs/asf001.sp2.ygridcore.net,49518,1365336962323/asf001.sp2.ygridcore.net%2C49518%2C1365336962323.1365336962362 2013-04-07 12:16:02,368 INFO [RegionServer:3;asf001.sp2.ygridcore.net,49518,1365336962323] wal.FSHLog(448): Using getNumCurrentReplicas--HDFS-826 2013-04-07 12:16:02,377 DEBUG [RegionServer:3;asf001.sp2.ygridcore.net,49518,1365336962323] executor.ExecutorService(99): Starting executor service name=RS_OPEN_REGION-asf001.sp2.ygridcore.net,49518,1365336962323, corePoolSize=3, maxPoolSize=3 2013-04-07 12:16:02,377 DEBUG [RegionServer:3;asf001.sp2.ygridcore.net,49518,1365336962323] executor.ExecutorService(99): Starting executor service name=RS_OPEN_META-asf001.sp2.ygridcore.net,49518,1365336962323, corePoolSize=1, maxPoolSize=1 2013-04-07 12:16:02,377 DEBUG [RegionServer:3;asf001.sp2.ygridcore.net,49518,1365336962323] executor.ExecutorService(99): Starting executor service name=RS_CLOSE_REGION-asf001.sp2.ygridcore.net,49518,1365336962323, corePoolSize=3, maxPoolSize=3 2013-04-07 12:16:02,378 DEBUG [RegionServer:3;asf001.sp2.ygridcore.net,49518,1365336962323] executor.ExecutorService(99): Starting executor service name=RS_CLOSE_META-asf001.sp2.ygridcore.net,49518,1365336962323, corePoolSize=1, maxPoolSize=1 2013-04-07 12:16:02,392 INFO [RegionServer:3;asf001.sp2.ygridcore.net,49518,1365336962323] regionserver.HRegionServer(1115): Serving as asf001.sp2.ygridcore.net,49518,1365336962323, RPC listening on asf001.sp2.ygridcore.net/67.195.138.31:49518, sessionid=0x13de46b5f8b0012 2013-04-07 12:16:02,392 INFO [SplitLogWorker-asf001.sp2.ygridcore.net,49518,1365336962323] regionserver.SplitLogWorker(135): SplitLogWorker asf001.sp2.ygridcore.net,49518,1365336962323 starting 2013-04-07 12:16:02,392 DEBUG [RegionServer:3;asf001.sp2.ygridcore.net,49518,1365336962323] procedure.ZKProcedureMemberRpcs(341): Starting procedure member 'asf001.sp2.ygridcore.net,49518,1365336962323' 2013-04-07 12:16:02,393 DEBUG [RegionServer:3;asf001.sp2.ygridcore.net,49518,1365336962323] procedure.ZKProcedureMemberRpcs(140): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2013-04-07 12:16:02,396 DEBUG [RegionServer:3;asf001.sp2.ygridcore.net,49518,1365336962323] procedure.ZKProcedureMemberRpcs(156): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2013-04-07 12:16:02,400 DEBUG [IPC Server handler 4 on 33796] zookeeper.ZKUtil(1680): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Retrieved 43 byte(s) of data from znode /hbase/meta-region-server and set watcher; asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:16:02,400 DEBUG [IPC Server handler 4 on 33796] master.TableLockManager$ZKTableLockManager$TableLockImpl(225): Attempt to acquire table write lock on :testRegionAssignmentAfterMasterRecoveryDueToZKExpiry for:C_M_CREATE_TABLE 2013-04-07 12:16:02,408 DEBUG [IPC Server handler 4 on 33796] zookeeper.ZKUtil(1680): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Retrieved 119 byte(s) of data from znode /hbase/table-lock/testRegionAssignmentAfterMasterRecoveryDueToZKExpiry/write-master:337960000000000; data=PBUF\x0A4testRegionAssignment... 2013-04-07 12:16:02,408 DEBUG [IPC Server handler 4 on 33796] lock.ZKInterProcessLockBase(220): Successfully acquired a lock for /hbase/table-lock/testRegionAssignmentAfterMasterRecoveryDueToZKExpiry/write-master:337960000000000 2013-04-07 12:16:02,408 DEBUG [IPC Server handler 4 on 33796] master.TableLockManager$ZKTableLockManager$TableLockImpl(246): Acquired table write lock on :testRegionAssignmentAfterMasterRecoveryDueToZKExpiry for:C_M_CREATE_TABLE 2013-04-07 12:16:02,409 DEBUG [IPC Server handler 4 on 33796] zookeeper.ZKUtil(1680): hconnection-0x1f327e-0x13de46b5f8b0003 Retrieved 43 byte(s) of data from znode /hbase/meta-region-server; data=asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:16:02,409 DEBUG [IPC Server handler 4 on 33796] client.ClientScanner(96): Creating scanner over .META. starting at key 'testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,,' 2013-04-07 12:16:02,409 DEBUG [IPC Server handler 4 on 33796] client.ClientScanner(209): Advancing internal scanner to startKey at 'testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,,' 2013-04-07 12:16:02,410 DEBUG [IPC Server handler 4 on 33796] zookeeper.ZKUtil(1680): hconnection-0x1f327e-0x13de46b5f8b0003 Retrieved 43 byte(s) of data from znode /hbase/meta-region-server; data=asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:16:02,414 DEBUG [IPC Server handler 4 on 33796] client.ClientScanner(196): Finished with scanning at {NAME => '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192,} 2013-04-07 12:16:02,418 INFO [MASTER_TABLE_OPERATIONS-asf001.sp2.ygridcore.net,33796,1365336878036-0] handler.CreateTableHandler(146): Attempting to create the table testRegionAssignmentAfterMasterRecoveryDueToZKExpiry 2013-04-07 12:16:02,418 DEBUG [pool-1-thread-1] client.MetaScanner(198): Scanning .META. starting at row=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,,00000000000000 for max=2147483647 rows using hconnection-0xfe04c6 2013-04-07 12:16:02,420 DEBUG [pool-1-thread-1] zookeeper.ZKUtil(1680): hconnection-0xfe04c6-0x13de46b5f8b0007 Retrieved 43 byte(s) of data from znode /hbase/meta-region-server; data=asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:16:02,441 INFO [RegionOpenAndInitThread-testRegionAssignmentAfterMasterRecoveryDueToZKExpiry-1] regionserver.HRegion(3862): creating HRegion testRegionAssignmentAfterMasterRecoveryDueToZKExpiry HTD == 'testRegionAssignmentAfterMasterRecoveryDueToZKExpiry' RootDir = hdfs://localhost:60405/user/jenkins/hbase/.tmp Table name == testRegionAssignmentAfterMasterRecoveryDueToZKExpiry 2013-04-07 12:16:02,441 INFO [RegionOpenAndInitThread-testRegionAssignmentAfterMasterRecoveryDueToZKExpiry-2] regionserver.HRegion(3862): creating HRegion testRegionAssignmentAfterMasterRecoveryDueToZKExpiry HTD == 'testRegionAssignmentAfterMasterRecoveryDueToZKExpiry' RootDir = hdfs://localhost:60405/user/jenkins/hbase/.tmp Table name == testRegionAssignmentAfterMasterRecoveryDueToZKExpiry 2013-04-07 12:16:02,441 INFO [RegionOpenAndInitThread-testRegionAssignmentAfterMasterRecoveryDueToZKExpiry-4] regionserver.HRegion(3862): creating HRegion testRegionAssignmentAfterMasterRecoveryDueToZKExpiry HTD == 'testRegionAssignmentAfterMasterRecoveryDueToZKExpiry' RootDir = hdfs://localhost:60405/user/jenkins/hbase/.tmp Table name == testRegionAssignmentAfterMasterRecoveryDueToZKExpiry 2013-04-07 12:16:02,442 INFO [RegionOpenAndInitThread-testRegionAssignmentAfterMasterRecoveryDueToZKExpiry-5] regionserver.HRegion(3862): creating HRegion testRegionAssignmentAfterMasterRecoveryDueToZKExpiry HTD == 'testRegionAssignmentAfterMasterRecoveryDueToZKExpiry' RootDir = hdfs://localhost:60405/user/jenkins/hbase/.tmp Table name == testRegionAssignmentAfterMasterRecoveryDueToZKExpiry 2013-04-07 12:16:02,441 INFO [RegionOpenAndInitThread-testRegionAssignmentAfterMasterRecoveryDueToZKExpiry-3] regionserver.HRegion(3862): creating HRegion testRegionAssignmentAfterMasterRecoveryDueToZKExpiry HTD == 'testRegionAssignmentAfterMasterRecoveryDueToZKExpiry' RootDir = hdfs://localhost:60405/user/jenkins/hbase/.tmp Table name == testRegionAssignmentAfterMasterRecoveryDueToZKExpiry 2013-04-07 12:16:02,442 INFO [RegionOpenAndInitThread-testRegionAssignmentAfterMasterRecoveryDueToZKExpiry-6] regionserver.HRegion(3862): creating HRegion testRegionAssignmentAfterMasterRecoveryDueToZKExpiry HTD == 'testRegionAssignmentAfterMasterRecoveryDueToZKExpiry' RootDir = hdfs://localhost:60405/user/jenkins/hbase/.tmp Table name == testRegionAssignmentAfterMasterRecoveryDueToZKExpiry 2013-04-07 12:16:02,443 INFO [RegionOpenAndInitThread-testRegionAssignmentAfterMasterRecoveryDueToZKExpiry-7] regionserver.HRegion(3862): creating HRegion testRegionAssignmentAfterMasterRecoveryDueToZKExpiry HTD == 'testRegionAssignmentAfterMasterRecoveryDueToZKExpiry' RootDir = hdfs://localhost:60405/user/jenkins/hbase/.tmp Table name == testRegionAssignmentAfterMasterRecoveryDueToZKExpiry 2013-04-07 12:16:02,443 INFO [RegionOpenAndInitThread-testRegionAssignmentAfterMasterRecoveryDueToZKExpiry-8] regionserver.HRegion(3862): creating HRegion testRegionAssignmentAfterMasterRecoveryDueToZKExpiry HTD == 'testRegionAssignmentAfterMasterRecoveryDueToZKExpiry' RootDir = hdfs://localhost:60405/user/jenkins/hbase/.tmp Table name == testRegionAssignmentAfterMasterRecoveryDueToZKExpiry 2013-04-07 12:16:02,443 INFO [RegionOpenAndInitThread-testRegionAssignmentAfterMasterRecoveryDueToZKExpiry-9] regionserver.HRegion(3862): creating HRegion testRegionAssignmentAfterMasterRecoveryDueToZKExpiry HTD == 'testRegionAssignmentAfterMasterRecoveryDueToZKExpiry' RootDir = hdfs://localhost:60405/user/jenkins/hbase/.tmp Table name == testRegionAssignmentAfterMasterRecoveryDueToZKExpiry 2013-04-07 12:16:02,444 INFO [RegionOpenAndInitThread-testRegionAssignmentAfterMasterRecoveryDueToZKExpiry-10] regionserver.HRegion(3862): creating HRegion testRegionAssignmentAfterMasterRecoveryDueToZKExpiry HTD == 'testRegionAssignmentAfterMasterRecoveryDueToZKExpiry' RootDir = hdfs://localhost:60405/user/jenkins/hbase/.tmp Table name == testRegionAssignmentAfterMasterRecoveryDueToZKExpiry 2013-04-07 12:16:02,447 DEBUG [RegionOpenAndInitThread-testRegionAssignmentAfterMasterRecoveryDueToZKExpiry-1] util.FSUtils(211): Creating file=hdfs://localhost:60405/user/jenkins/hbase/.tmp/testRegionAssignmentAfterMasterRecoveryDueToZKExpiry/d4e129e51e67d798e17083623fdde133/.regioninfo with permission=rwxrwxrwx 2013-04-07 12:16:02,448 DEBUG [RegionOpenAndInitThread-testRegionAssignmentAfterMasterRecoveryDueToZKExpiry-4] util.FSUtils(211): Creating file=hdfs://localhost:60405/user/jenkins/hbase/.tmp/testRegionAssignmentAfterMasterRecoveryDueToZKExpiry/4432e800611c00a4ae168862a273fc70/.regioninfo with permission=rwxrwxrwx 2013-04-07 12:16:02,449 DEBUG [RegionOpenAndInitThread-testRegionAssignmentAfterMasterRecoveryDueToZKExpiry-7] util.FSUtils(211): Creating file=hdfs://localhost:60405/user/jenkins/hbase/.tmp/testRegionAssignmentAfterMasterRecoveryDueToZKExpiry/eff8376859f5e6ba3201ec0ec9536d0f/.regioninfo with permission=rwxrwxrwx 2013-04-07 12:16:02,449 DEBUG [RegionOpenAndInitThread-testRegionAssignmentAfterMasterRecoveryDueToZKExpiry-2] util.FSUtils(211): Creating file=hdfs://localhost:60405/user/jenkins/hbase/.tmp/testRegionAssignmentAfterMasterRecoveryDueToZKExpiry/b1b340dff1e06fd6cb0b0a9c64964551/.regioninfo with permission=rwxrwxrwx 2013-04-07 12:16:02,451 DEBUG [RegionOpenAndInitThread-testRegionAssignmentAfterMasterRecoveryDueToZKExpiry-6] util.FSUtils(211): Creating file=hdfs://localhost:60405/user/jenkins/hbase/.tmp/testRegionAssignmentAfterMasterRecoveryDueToZKExpiry/dbb82f7f432d6664327da0b8f15b7d76/.regioninfo with permission=rwxrwxrwx 2013-04-07 12:16:02,451 DEBUG [RegionOpenAndInitThread-testRegionAssignmentAfterMasterRecoveryDueToZKExpiry-8] util.FSUtils(211): Creating file=hdfs://localhost:60405/user/jenkins/hbase/.tmp/testRegionAssignmentAfterMasterRecoveryDueToZKExpiry/682475e1358b7306bd3406e9330263a4/.regioninfo with permission=rwxrwxrwx 2013-04-07 12:16:02,451 DEBUG [RegionOpenAndInitThread-testRegionAssignmentAfterMasterRecoveryDueToZKExpiry-3] util.FSUtils(211): Creating file=hdfs://localhost:60405/user/jenkins/hbase/.tmp/testRegionAssignmentAfterMasterRecoveryDueToZKExpiry/11c8123e8d563f264d947ffa106e4c7c/.regioninfo with permission=rwxrwxrwx 2013-04-07 12:16:02,452 DEBUG [RegionOpenAndInitThread-testRegionAssignmentAfterMasterRecoveryDueToZKExpiry-5] util.FSUtils(211): Creating file=hdfs://localhost:60405/user/jenkins/hbase/.tmp/testRegionAssignmentAfterMasterRecoveryDueToZKExpiry/e99798d2c4d566ddb063efe03842457f/.regioninfo with permission=rwxrwxrwx 2013-04-07 12:16:02,453 DEBUG [RegionOpenAndInitThread-testRegionAssignmentAfterMasterRecoveryDueToZKExpiry-10] util.FSUtils(211): Creating file=hdfs://localhost:60405/user/jenkins/hbase/.tmp/testRegionAssignmentAfterMasterRecoveryDueToZKExpiry/385e5846630b0cb4493758961a2c2457/.regioninfo with permission=rwxrwxrwx 2013-04-07 12:16:02,454 DEBUG [RegionOpenAndInitThread-testRegionAssignmentAfterMasterRecoveryDueToZKExpiry-9] util.FSUtils(211): Creating file=hdfs://localhost:60405/user/jenkins/hbase/.tmp/testRegionAssignmentAfterMasterRecoveryDueToZKExpiry/1ce6d62427641e2a0e135d3d27b54b5d/.regioninfo with permission=rwxrwxrwx 2013-04-07 12:16:02,495 DEBUG [RegionOpenAndInitThread-testRegionAssignmentAfterMasterRecoveryDueToZKExpiry-6] regionserver.HRegion(496): Instantiated testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,e,1365336962398.dbb82f7f432d6664327da0b8f15b7d76. 2013-04-07 12:16:02,495 DEBUG [RegionOpenAndInitThread-testRegionAssignmentAfterMasterRecoveryDueToZKExpiry-6] regionserver.HRegion(889): Closing testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,e,1365336962398.dbb82f7f432d6664327da0b8f15b7d76.: disabling compactions & flushes 2013-04-07 12:16:02,495 DEBUG [RegionOpenAndInitThread-testRegionAssignmentAfterMasterRecoveryDueToZKExpiry-6] regionserver.HRegion(911): Updates disabled for region testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,e,1365336962398.dbb82f7f432d6664327da0b8f15b7d76. 2013-04-07 12:16:02,495 INFO [RegionOpenAndInitThread-testRegionAssignmentAfterMasterRecoveryDueToZKExpiry-6] regionserver.HRegion(968): Closed testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,e,1365336962398.dbb82f7f432d6664327da0b8f15b7d76. 2013-04-07 12:16:02,495 INFO [RegionOpenAndInitThread-testRegionAssignmentAfterMasterRecoveryDueToZKExpiry-6] regionserver.HRegion(3862): creating HRegion testRegionAssignmentAfterMasterRecoveryDueToZKExpiry HTD == 'testRegionAssignmentAfterMasterRecoveryDueToZKExpiry' RootDir = hdfs://localhost:60405/user/jenkins/hbase/.tmp Table name == testRegionAssignmentAfterMasterRecoveryDueToZKExpiry 2013-04-07 12:16:02,504 DEBUG [RegionOpenAndInitThread-testRegionAssignmentAfterMasterRecoveryDueToZKExpiry-5] regionserver.HRegion(496): Instantiated testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,d,1365336962398.e99798d2c4d566ddb063efe03842457f. 2013-04-07 12:16:02,504 DEBUG [RegionOpenAndInitThread-testRegionAssignmentAfterMasterRecoveryDueToZKExpiry-5] regionserver.HRegion(889): Closing testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,d,1365336962398.e99798d2c4d566ddb063efe03842457f.: disabling compactions & flushes 2013-04-07 12:16:02,504 DEBUG [RegionOpenAndInitThread-testRegionAssignmentAfterMasterRecoveryDueToZKExpiry-5] regionserver.HRegion(911): Updates disabled for region testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,d,1365336962398.e99798d2c4d566ddb063efe03842457f. 2013-04-07 12:16:02,504 INFO [RegionOpenAndInitThread-testRegionAssignmentAfterMasterRecoveryDueToZKExpiry-5] regionserver.HRegion(968): Closed testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,d,1365336962398.e99798d2c4d566ddb063efe03842457f. 2013-04-07 12:16:02,505 DEBUG [RegionOpenAndInitThread-testRegionAssignmentAfterMasterRecoveryDueToZKExpiry-4] regionserver.HRegion(496): Instantiated testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,c,1365336962398.4432e800611c00a4ae168862a273fc70. 2013-04-07 12:16:02,507 DEBUG [RegionOpenAndInitThread-testRegionAssignmentAfterMasterRecoveryDueToZKExpiry-4] regionserver.HRegion(889): Closing testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,c,1365336962398.4432e800611c00a4ae168862a273fc70.: disabling compactions & flushes 2013-04-07 12:16:02,507 DEBUG [RegionOpenAndInitThread-testRegionAssignmentAfterMasterRecoveryDueToZKExpiry-4] regionserver.HRegion(911): Updates disabled for region testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,c,1365336962398.4432e800611c00a4ae168862a273fc70. 2013-04-07 12:16:02,507 INFO [RegionOpenAndInitThread-testRegionAssignmentAfterMasterRecoveryDueToZKExpiry-4] regionserver.HRegion(968): Closed testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,c,1365336962398.4432e800611c00a4ae168862a273fc70. 2013-04-07 12:16:02,512 DEBUG [RegionOpenAndInitThread-testRegionAssignmentAfterMasterRecoveryDueToZKExpiry-7] regionserver.HRegion(496): Instantiated testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,f,1365336962398.eff8376859f5e6ba3201ec0ec9536d0f. 2013-04-07 12:16:02,512 DEBUG [RegionOpenAndInitThread-testRegionAssignmentAfterMasterRecoveryDueToZKExpiry-7] regionserver.HRegion(889): Closing testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,f,1365336962398.eff8376859f5e6ba3201ec0ec9536d0f.: disabling compactions & flushes 2013-04-07 12:16:02,513 DEBUG [RegionOpenAndInitThread-testRegionAssignmentAfterMasterRecoveryDueToZKExpiry-7] regionserver.HRegion(911): Updates disabled for region testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,f,1365336962398.eff8376859f5e6ba3201ec0ec9536d0f. 2013-04-07 12:16:02,513 INFO [RegionOpenAndInitThread-testRegionAssignmentAfterMasterRecoveryDueToZKExpiry-7] regionserver.HRegion(968): Closed testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,f,1365336962398.eff8376859f5e6ba3201ec0ec9536d0f. 2013-04-07 12:16:02,512 DEBUG [RegionOpenAndInitThread-testRegionAssignmentAfterMasterRecoveryDueToZKExpiry-6] util.FSUtils(211): Creating file=hdfs://localhost:60405/user/jenkins/hbase/.tmp/testRegionAssignmentAfterMasterRecoveryDueToZKExpiry/38f0eb146fb473b19541deaada3419b9/.regioninfo with permission=rwxrwxrwx 2013-04-07 12:16:02,516 DEBUG [RegionOpenAndInitThread-testRegionAssignmentAfterMasterRecoveryDueToZKExpiry-8] regionserver.HRegion(496): Instantiated testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,g,1365336962398.682475e1358b7306bd3406e9330263a4. 2013-04-07 12:16:02,517 DEBUG [RegionOpenAndInitThread-testRegionAssignmentAfterMasterRecoveryDueToZKExpiry-8] regionserver.HRegion(889): Closing testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,g,1365336962398.682475e1358b7306bd3406e9330263a4.: disabling compactions & flushes 2013-04-07 12:16:02,517 DEBUG [RegionOpenAndInitThread-testRegionAssignmentAfterMasterRecoveryDueToZKExpiry-8] regionserver.HRegion(911): Updates disabled for region testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,g,1365336962398.682475e1358b7306bd3406e9330263a4. 2013-04-07 12:16:02,517 INFO [RegionOpenAndInitThread-testRegionAssignmentAfterMasterRecoveryDueToZKExpiry-8] regionserver.HRegion(968): Closed testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,g,1365336962398.682475e1358b7306bd3406e9330263a4. 2013-04-07 12:16:02,517 DEBUG [RegionOpenAndInitThread-testRegionAssignmentAfterMasterRecoveryDueToZKExpiry-1] regionserver.HRegion(496): Instantiated testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,,1365336962397.d4e129e51e67d798e17083623fdde133. 2013-04-07 12:16:02,517 DEBUG [RegionOpenAndInitThread-testRegionAssignmentAfterMasterRecoveryDueToZKExpiry-1] regionserver.HRegion(889): Closing testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,,1365336962397.d4e129e51e67d798e17083623fdde133.: disabling compactions & flushes 2013-04-07 12:16:02,517 DEBUG [RegionOpenAndInitThread-testRegionAssignmentAfterMasterRecoveryDueToZKExpiry-1] regionserver.HRegion(911): Updates disabled for region testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,,1365336962397.d4e129e51e67d798e17083623fdde133. 2013-04-07 12:16:02,518 INFO [RegionOpenAndInitThread-testRegionAssignmentAfterMasterRecoveryDueToZKExpiry-1] regionserver.HRegion(968): Closed testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,,1365336962397.d4e129e51e67d798e17083623fdde133. 2013-04-07 12:16:02,527 DEBUG [RegionOpenAndInitThread-testRegionAssignmentAfterMasterRecoveryDueToZKExpiry-10] regionserver.HRegion(496): Instantiated testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,i,1365336962398.385e5846630b0cb4493758961a2c2457. 2013-04-07 12:16:02,527 DEBUG [RegionOpenAndInitThread-testRegionAssignmentAfterMasterRecoveryDueToZKExpiry-10] regionserver.HRegion(889): Closing testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,i,1365336962398.385e5846630b0cb4493758961a2c2457.: disabling compactions & flushes 2013-04-07 12:16:02,527 DEBUG [RegionOpenAndInitThread-testRegionAssignmentAfterMasterRecoveryDueToZKExpiry-10] regionserver.HRegion(911): Updates disabled for region testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,i,1365336962398.385e5846630b0cb4493758961a2c2457. 2013-04-07 12:16:02,527 INFO [RegionOpenAndInitThread-testRegionAssignmentAfterMasterRecoveryDueToZKExpiry-10] regionserver.HRegion(968): Closed testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,i,1365336962398.385e5846630b0cb4493758961a2c2457. 2013-04-07 12:16:02,908 DEBUG [RegionOpenAndInitThread-testRegionAssignmentAfterMasterRecoveryDueToZKExpiry-2] regionserver.HRegion(496): Instantiated testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,a,1365336962398.b1b340dff1e06fd6cb0b0a9c64964551. 2013-04-07 12:16:02,908 DEBUG [RegionOpenAndInitThread-testRegionAssignmentAfterMasterRecoveryDueToZKExpiry-2] regionserver.HRegion(889): Closing testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,a,1365336962398.b1b340dff1e06fd6cb0b0a9c64964551.: disabling compactions & flushes 2013-04-07 12:16:02,909 DEBUG [RegionOpenAndInitThread-testRegionAssignmentAfterMasterRecoveryDueToZKExpiry-2] regionserver.HRegion(911): Updates disabled for region testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,a,1365336962398.b1b340dff1e06fd6cb0b0a9c64964551. 2013-04-07 12:16:02,909 INFO [RegionOpenAndInitThread-testRegionAssignmentAfterMasterRecoveryDueToZKExpiry-2] regionserver.HRegion(968): Closed testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,a,1365336962398.b1b340dff1e06fd6cb0b0a9c64964551. 2013-04-07 12:16:02,915 DEBUG [RegionOpenAndInitThread-testRegionAssignmentAfterMasterRecoveryDueToZKExpiry-9] regionserver.HRegion(496): Instantiated testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,h,1365336962398.1ce6d62427641e2a0e135d3d27b54b5d. 2013-04-07 12:16:02,915 DEBUG [RegionOpenAndInitThread-testRegionAssignmentAfterMasterRecoveryDueToZKExpiry-9] regionserver.HRegion(889): Closing testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,h,1365336962398.1ce6d62427641e2a0e135d3d27b54b5d.: disabling compactions & flushes 2013-04-07 12:16:02,915 DEBUG [RegionOpenAndInitThread-testRegionAssignmentAfterMasterRecoveryDueToZKExpiry-9] regionserver.HRegion(911): Updates disabled for region testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,h,1365336962398.1ce6d62427641e2a0e135d3d27b54b5d. 2013-04-07 12:16:02,916 INFO [RegionOpenAndInitThread-testRegionAssignmentAfterMasterRecoveryDueToZKExpiry-9] regionserver.HRegion(968): Closed testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,h,1365336962398.1ce6d62427641e2a0e135d3d27b54b5d. 2013-04-07 12:16:02,917 DEBUG [RegionOpenAndInitThread-testRegionAssignmentAfterMasterRecoveryDueToZKExpiry-3] regionserver.HRegion(496): Instantiated testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,b,1365336962398.11c8123e8d563f264d947ffa106e4c7c. 2013-04-07 12:16:02,918 DEBUG [RegionOpenAndInitThread-testRegionAssignmentAfterMasterRecoveryDueToZKExpiry-3] regionserver.HRegion(889): Closing testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,b,1365336962398.11c8123e8d563f264d947ffa106e4c7c.: disabling compactions & flushes 2013-04-07 12:16:02,918 DEBUG [RegionOpenAndInitThread-testRegionAssignmentAfterMasterRecoveryDueToZKExpiry-3] regionserver.HRegion(911): Updates disabled for region testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,b,1365336962398.11c8123e8d563f264d947ffa106e4c7c. 2013-04-07 12:16:02,918 INFO [RegionOpenAndInitThread-testRegionAssignmentAfterMasterRecoveryDueToZKExpiry-3] regionserver.HRegion(968): Closed testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,b,1365336962398.11c8123e8d563f264d947ffa106e4c7c. 2013-04-07 12:16:02,969 DEBUG [RegionOpenAndInitThread-testRegionAssignmentAfterMasterRecoveryDueToZKExpiry-6] regionserver.HRegion(496): Instantiated testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,j,1365336962398.38f0eb146fb473b19541deaada3419b9. 2013-04-07 12:16:02,969 DEBUG [RegionOpenAndInitThread-testRegionAssignmentAfterMasterRecoveryDueToZKExpiry-6] regionserver.HRegion(889): Closing testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,j,1365336962398.38f0eb146fb473b19541deaada3419b9.: disabling compactions & flushes 2013-04-07 12:16:02,969 DEBUG [RegionOpenAndInitThread-testRegionAssignmentAfterMasterRecoveryDueToZKExpiry-6] regionserver.HRegion(911): Updates disabled for region testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,j,1365336962398.38f0eb146fb473b19541deaada3419b9. 2013-04-07 12:16:02,969 INFO [RegionOpenAndInitThread-testRegionAssignmentAfterMasterRecoveryDueToZKExpiry-6] regionserver.HRegion(968): Closed testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,j,1365336962398.38f0eb146fb473b19541deaada3419b9. 2013-04-07 12:16:02,973 DEBUG [MASTER_TABLE_OPERATIONS-asf001.sp2.ygridcore.net,33796,1365336878036-0] zookeeper.ZKUtil(1680): hconnection-0x1f327e-0x13de46b5f8b0003 Retrieved 43 byte(s) of data from znode /hbase/meta-region-server; data=asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:16:02,974 DEBUG [MASTER_TABLE_OPERATIONS-asf001.sp2.ygridcore.net,33796,1365336878036-0] zookeeper.ZKUtil(1680): hconnection-0x1f327e-0x13de46b5f8b0003 Retrieved 43 byte(s) of data from znode /hbase/meta-region-server; data=asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:16:02,975 DEBUG [MASTER_TABLE_OPERATIONS-asf001.sp2.ygridcore.net,33796,1365336878036-0] zookeeper.ZKUtil(1680): hconnection-0x1f327e-0x13de46b5f8b0003 Retrieved 43 byte(s) of data from znode /hbase/meta-region-server; data=asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:16:02,976 DEBUG [MASTER_TABLE_OPERATIONS-asf001.sp2.ygridcore.net,33796,1365336878036-0] zookeeper.ZKUtil(1680): hconnection-0x1f327e-0x13de46b5f8b0003 Retrieved 43 byte(s) of data from znode /hbase/meta-region-server; data=asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:16:02,977 DEBUG [MASTER_TABLE_OPERATIONS-asf001.sp2.ygridcore.net,33796,1365336878036-0] zookeeper.ZKUtil(1680): hconnection-0x1f327e-0x13de46b5f8b0003 Retrieved 43 byte(s) of data from znode /hbase/meta-region-server; data=asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:16:02,978 DEBUG [MASTER_TABLE_OPERATIONS-asf001.sp2.ygridcore.net,33796,1365336878036-0] zookeeper.ZKUtil(1680): hconnection-0x1f327e-0x13de46b5f8b0003 Retrieved 43 byte(s) of data from znode /hbase/meta-region-server; data=asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:16:02,979 DEBUG [MASTER_TABLE_OPERATIONS-asf001.sp2.ygridcore.net,33796,1365336878036-0] zookeeper.ZKUtil(1680): hconnection-0x1f327e-0x13de46b5f8b0003 Retrieved 43 byte(s) of data from znode /hbase/meta-region-server; data=asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:16:02,980 DEBUG [MASTER_TABLE_OPERATIONS-asf001.sp2.ygridcore.net,33796,1365336878036-0] zookeeper.ZKUtil(1680): hconnection-0x1f327e-0x13de46b5f8b0003 Retrieved 43 byte(s) of data from znode /hbase/meta-region-server; data=asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:16:02,981 DEBUG [MASTER_TABLE_OPERATIONS-asf001.sp2.ygridcore.net,33796,1365336878036-0] zookeeper.ZKUtil(1680): hconnection-0x1f327e-0x13de46b5f8b0003 Retrieved 43 byte(s) of data from znode /hbase/meta-region-server; data=asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:16:02,982 DEBUG [MASTER_TABLE_OPERATIONS-asf001.sp2.ygridcore.net,33796,1365336878036-0] zookeeper.ZKUtil(1680): hconnection-0x1f327e-0x13de46b5f8b0003 Retrieved 43 byte(s) of data from znode /hbase/meta-region-server; data=asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:16:02,983 DEBUG [MASTER_TABLE_OPERATIONS-asf001.sp2.ygridcore.net,33796,1365336878036-0] zookeeper.ZKUtil(1680): hconnection-0x1f327e-0x13de46b5f8b0003 Retrieved 43 byte(s) of data from znode /hbase/meta-region-server; data=asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:16:02,983 DEBUG [MASTER_TABLE_OPERATIONS-asf001.sp2.ygridcore.net,33796,1365336878036-0] zookeeper.ZKUtil(1680): hconnection-0x1f327e-0x13de46b5f8b0003 Retrieved 43 byte(s) of data from znode /hbase/meta-region-server; data=asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:16:02,988 INFO [MASTER_TABLE_OPERATIONS-asf001.sp2.ygridcore.net,33796,1365336878036-0] catalog.MetaEditor(254): Added 11 regions in META 2013-04-07 12:16:02,990 INFO [MASTER_TABLE_OPERATIONS-asf001.sp2.ygridcore.net,33796,1365336878036-0] master.AssignmentManager(2370): Bulk assigning 11 region(s) across 3 server(s), round-robin=true 2013-04-07 12:16:02,992 DEBUG [asf001.sp2.ygridcore.net,33796,1365336878036-GeneralBulkAssigner-0] master.AssignmentManager(1444): Bulk assigning 3 region(s) to asf001.sp2.ygridcore.net,43802,1365336889418 2013-04-07 12:16:02,992 DEBUG [asf001.sp2.ygridcore.net,33796,1365336878036-GeneralBulkAssigner-1] master.AssignmentManager(1444): Bulk assigning 4 region(s) to asf001.sp2.ygridcore.net,49518,1365336962323 2013-04-07 12:16:02,992 INFO [asf001.sp2.ygridcore.net,33796,1365336878036-GeneralBulkAssigner-0] master.RegionStates(264): Region {NAME => 'testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,c,1365336962398.4432e800611c00a4ae168862a273fc70.', STARTKEY => 'c', ENDKEY => 'd', ENCODED => 4432e800611c00a4ae168862a273fc70,} transitioned from {testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,c,1365336962398.4432e800611c00a4ae168862a273fc70. state=OFFLINE, ts=1365336962989, server=null} to {testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,c,1365336962398.4432e800611c00a4ae168862a273fc70. state=OFFLINE, ts=1365336962992, server=null} 2013-04-07 12:16:02,992 DEBUG [MASTER_TABLE_OPERATIONS-asf001.sp2.ygridcore.net,33796,1365336878036-0] master.GeneralBulkAssigner(177): Timeout-on-RIT=154000 2013-04-07 12:16:02,992 INFO [asf001.sp2.ygridcore.net,33796,1365336878036-GeneralBulkAssigner-1] master.RegionStates(264): Region {NAME => 'testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,d,1365336962398.e99798d2c4d566ddb063efe03842457f.', STARTKEY => 'd', ENDKEY => 'e', ENCODED => e99798d2c4d566ddb063efe03842457f,} transitioned from {testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,d,1365336962398.e99798d2c4d566ddb063efe03842457f. state=OFFLINE, ts=1365336962989, server=null} to {testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,d,1365336962398.e99798d2c4d566ddb063efe03842457f. state=OFFLINE, ts=1365336962992, server=null} 2013-04-07 12:16:02,992 DEBUG [asf001.sp2.ygridcore.net,33796,1365336878036-GeneralBulkAssigner-2] master.AssignmentManager(1444): Bulk assigning 4 region(s) to asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:16:02,992 DEBUG [asf001.sp2.ygridcore.net,33796,1365336878036-GeneralBulkAssigner-1] zookeeper.ZKAssign(177): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Async create of unassigned node for e99798d2c4d566ddb063efe03842457f with OFFLINE state 2013-04-07 12:16:02,992 DEBUG [asf001.sp2.ygridcore.net,33796,1365336878036-GeneralBulkAssigner-0] zookeeper.ZKAssign(177): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Async create of unassigned node for 4432e800611c00a4ae168862a273fc70 with OFFLINE state 2013-04-07 12:16:02,993 INFO [asf001.sp2.ygridcore.net,33796,1365336878036-GeneralBulkAssigner-2] master.RegionStates(264): Region {NAME => 'testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,e,1365336962398.dbb82f7f432d6664327da0b8f15b7d76.', STARTKEY => 'e', ENDKEY => 'f', ENCODED => dbb82f7f432d6664327da0b8f15b7d76,} transitioned from {testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,e,1365336962398.dbb82f7f432d6664327da0b8f15b7d76. state=OFFLINE, ts=1365336962989, server=null} to {testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,e,1365336962398.dbb82f7f432d6664327da0b8f15b7d76. state=OFFLINE, ts=1365336962993, server=null} 2013-04-07 12:16:02,994 DEBUG [asf001.sp2.ygridcore.net,33796,1365336878036-GeneralBulkAssigner-2] zookeeper.ZKAssign(177): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Async create of unassigned node for dbb82f7f432d6664327da0b8f15b7d76 with OFFLINE state 2013-04-07 12:16:02,994 INFO [asf001.sp2.ygridcore.net,33796,1365336878036-GeneralBulkAssigner-0] master.RegionStates(264): Region {NAME => 'testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,,1365336962397.d4e129e51e67d798e17083623fdde133.', STARTKEY => '', ENDKEY => 'a', ENCODED => d4e129e51e67d798e17083623fdde133,} transitioned from {testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,,1365336962397.d4e129e51e67d798e17083623fdde133. state=OFFLINE, ts=1365336962989, server=null} to {testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,,1365336962397.d4e129e51e67d798e17083623fdde133. state=OFFLINE, ts=1365336962994, server=null} 2013-04-07 12:16:02,994 DEBUG [asf001.sp2.ygridcore.net,33796,1365336878036-GeneralBulkAssigner-0] zookeeper.ZKAssign(177): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Async create of unassigned node for d4e129e51e67d798e17083623fdde133 with OFFLINE state 2013-04-07 12:16:02,994 INFO [asf001.sp2.ygridcore.net,33796,1365336878036-GeneralBulkAssigner-2] master.RegionStates(264): Region {NAME => 'testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,f,1365336962398.eff8376859f5e6ba3201ec0ec9536d0f.', STARTKEY => 'f', ENDKEY => 'g', ENCODED => eff8376859f5e6ba3201ec0ec9536d0f,} transitioned from {testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,f,1365336962398.eff8376859f5e6ba3201ec0ec9536d0f. state=OFFLINE, ts=1365336962989, server=null} to {testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,f,1365336962398.eff8376859f5e6ba3201ec0ec9536d0f. state=OFFLINE, ts=1365336962994, server=null} 2013-04-07 12:16:02,994 DEBUG [asf001.sp2.ygridcore.net,33796,1365336878036-GeneralBulkAssigner-2] zookeeper.ZKAssign(177): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Async create of unassigned node for eff8376859f5e6ba3201ec0ec9536d0f with OFFLINE state 2013-04-07 12:16:02,994 INFO [asf001.sp2.ygridcore.net,33796,1365336878036-GeneralBulkAssigner-0] master.RegionStates(264): Region {NAME => 'testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,h,1365336962398.1ce6d62427641e2a0e135d3d27b54b5d.', STARTKEY => 'h', ENDKEY => 'i', ENCODED => 1ce6d62427641e2a0e135d3d27b54b5d,} transitioned from {testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,h,1365336962398.1ce6d62427641e2a0e135d3d27b54b5d. state=OFFLINE, ts=1365336962989, server=null} to {testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,h,1365336962398.1ce6d62427641e2a0e135d3d27b54b5d. state=OFFLINE, ts=1365336962994, server=null} 2013-04-07 12:16:03,027 INFO [PRI IPC Server handler 0 on 49518] regionserver.HRegionServer(3346): Received request to open region: testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,d,1365336962398.e99798d2c4d566ddb063efe03842457f. on asf001.sp2.ygridcore.net,49518,1365336962323 2013-04-07 12:16:03,042 INFO [PRI IPC Server handler 0 on 49518] regionserver.HRegionServer(3346): Received request to open region: testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,g,1365336962398.682475e1358b7306bd3406e9330263a4. on asf001.sp2.ygridcore.net,49518,1365336962323 2013-04-07 12:16:03,043 DEBUG [RS_OPEN_REGION-asf001.sp2.ygridcore.net,60884,1365336878389-1] zookeeper.ZKAssign(721): regionserver:60884-0x13de46b5f8b0001 Attempting to transition node eff8376859f5e6ba3201ec0ec9536d0f from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING 2013-04-07 12:16:03,043 DEBUG [RS_OPEN_REGION-asf001.sp2.ygridcore.net,60884,1365336878389-0] zookeeper.ZKUtil(1680): regionserver:60884-0x13de46b5f8b0001 Retrieved 156 byte(s) of data from znode /hbase/region-in-transition/dbb82f7f432d6664327da0b8f15b7d76; data=region=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,e,1365336962398.dbb82f7f432d6664327da0b8f15b7d76., state=M_ZK_REGION_OFFLINE, servername=asf001.sp2.ygridcore.net,60884,1365336878389, createTime=1365336962994, payload.length=0 2013-04-07 12:16:03,044 INFO [PRI IPC Server handler 0 on 49518] regionserver.HRegionServer(3346): Received request to open region: testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,a,1365336962398.b1b340dff1e06fd6cb0b0a9c64964551. on asf001.sp2.ygridcore.net,49518,1365336962323 2013-04-07 12:16:03,045 INFO [PRI IPC Server handler 0 on 49518] regionserver.HRegionServer(3346): Received request to open region: testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,j,1365336962398.38f0eb146fb473b19541deaada3419b9. on asf001.sp2.ygridcore.net,49518,1365336962323 2013-04-07 12:16:03,046 DEBUG [RS_OPEN_REGION-asf001.sp2.ygridcore.net,60884,1365336878389-2] zookeeper.ZKAssign(721): regionserver:60884-0x13de46b5f8b0001 Attempting to transition node 385e5846630b0cb4493758961a2c2457 from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING 2013-04-07 12:16:03,046 DEBUG [RS_OPEN_REGION-asf001.sp2.ygridcore.net,49518,1365336962323-0] zookeeper.ZKAssign(721): regionserver:49518-0x13de46b5f8b0012 Attempting to transition node e99798d2c4d566ddb063efe03842457f from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING 2013-04-07 12:16:03,048 DEBUG [asf001.sp2.ygridcore.net,33796,1365336878036-GeneralBulkAssigner-1] master.AssignmentManager(1600): Bulk assigning done for asf001.sp2.ygridcore.net,49518,1365336962323 2013-04-07 12:16:03,048 DEBUG [RS_OPEN_REGION-asf001.sp2.ygridcore.net,49518,1365336962323-2] zookeeper.ZKAssign(721): regionserver:49518-0x13de46b5f8b0012 Attempting to transition node b1b340dff1e06fd6cb0b0a9c64964551 from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING 2013-04-07 12:16:03,049 DEBUG [RS_OPEN_REGION-asf001.sp2.ygridcore.net,49518,1365336962323-1] zookeeper.ZKAssign(721): regionserver:49518-0x13de46b5f8b0012 Attempting to transition node 682475e1358b7306bd3406e9330263a4 from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING 2013-04-07 12:16:03,052 DEBUG [RS_OPEN_REGION-asf001.sp2.ygridcore.net,49518,1365336962323-0] zookeeper.ZKUtil(1680): regionserver:49518-0x13de46b5f8b0012 Retrieved 156 byte(s) of data from znode /hbase/region-in-transition/e99798d2c4d566ddb063efe03842457f; data=region=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,d,1365336962398.e99798d2c4d566ddb063efe03842457f., state=M_ZK_REGION_OFFLINE, servername=asf001.sp2.ygridcore.net,49518,1365336962323, createTime=1365336962993, payload.length=0 2013-04-07 12:16:03,053 DEBUG [RS_OPEN_REGION-asf001.sp2.ygridcore.net,49518,1365336962323-1] zookeeper.ZKUtil(1680): regionserver:49518-0x13de46b5f8b0012 Retrieved 156 byte(s) of data from znode /hbase/region-in-transition/682475e1358b7306bd3406e9330263a4; data=region=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,g,1365336962398.682475e1358b7306bd3406e9330263a4., state=M_ZK_REGION_OFFLINE, servername=asf001.sp2.ygridcore.net,49518,1365336962323, createTime=1365336962995, payload.length=0 2013-04-07 12:16:03,056 DEBUG [RS_OPEN_REGION-asf001.sp2.ygridcore.net,49518,1365336962323-2] zookeeper.ZKUtil(1680): regionserver:49518-0x13de46b5f8b0012 Retrieved 156 byte(s) of data from znode /hbase/region-in-transition/b1b340dff1e06fd6cb0b0a9c64964551; data=region=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,a,1365336962398.b1b340dff1e06fd6cb0b0a9c64964551., state=M_ZK_REGION_OFFLINE, servername=asf001.sp2.ygridcore.net,49518,1365336962323, createTime=1365336962995, payload.length=0 2013-04-07 12:16:03,058 DEBUG [RS_OPEN_REGION-asf001.sp2.ygridcore.net,49518,1365336962323-0] zookeeper.ZKAssign(784): regionserver:49518-0x13de46b5f8b0012 Successfully transitioned node e99798d2c4d566ddb063efe03842457f from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING 2013-04-07 12:16:03,058 INFO [RS_OPEN_REGION-asf001.sp2.ygridcore.net,49518,1365336962323-0] regionserver.HRegion(4023): HRegion.openHRegion Region name ==testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,d,1365336962398.e99798d2c4d566ddb063efe03842457f. 2013-04-07 12:16:03,058 DEBUG [RS_OPEN_REGION-asf001.sp2.ygridcore.net,49518,1365336962323-0] regionserver.HRegion(4025): Opening region: {NAME => 'testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,d,1365336962398.e99798d2c4d566ddb063efe03842457f.', STARTKEY => 'd', ENDKEY => 'e', ENCODED => e99798d2c4d566ddb063efe03842457f,} 2013-04-07 12:16:03,060 DEBUG [RS_OPEN_REGION-asf001.sp2.ygridcore.net,49518,1365336962323-0] regionserver.MetricsRegionSourceImpl(54): Creating new MetricsRegionSourceImpl for table testRegionAssignmentAfterMasterRecoveryDueToZKExpiry e99798d2c4d566ddb063efe03842457f 2013-04-07 12:16:03,060 DEBUG [RS_OPEN_REGION-asf001.sp2.ygridcore.net,49518,1365336962323-0] regionserver.HRegion(496): Instantiated testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,d,1365336962398.e99798d2c4d566ddb063efe03842457f. 2013-04-07 12:16:03,062 DEBUG [RS_OPEN_REGION-asf001.sp2.ygridcore.net,49518,1365336962323-1] zookeeper.ZKAssign(784): regionserver:49518-0x13de46b5f8b0012 Successfully transitioned node 682475e1358b7306bd3406e9330263a4 from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING 2013-04-07 12:16:03,062 INFO [RS_OPEN_REGION-asf001.sp2.ygridcore.net,49518,1365336962323-1] regionserver.HRegion(4023): HRegion.openHRegion Region name ==testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,g,1365336962398.682475e1358b7306bd3406e9330263a4. 2013-04-07 12:16:03,063 DEBUG [RS_OPEN_REGION-asf001.sp2.ygridcore.net,49518,1365336962323-1] regionserver.HRegion(4025): Opening region: {NAME => 'testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,g,1365336962398.682475e1358b7306bd3406e9330263a4.', STARTKEY => 'g', ENDKEY => 'h', ENCODED => 682475e1358b7306bd3406e9330263a4,} 2013-04-07 12:16:03,065 DEBUG [RS_OPEN_REGION-asf001.sp2.ygridcore.net,49518,1365336962323-1] regionserver.MetricsRegionSourceImpl(54): Creating new MetricsRegionSourceImpl for table testRegionAssignmentAfterMasterRecoveryDueToZKExpiry 682475e1358b7306bd3406e9330263a4 2013-04-07 12:16:03,065 DEBUG [RS_OPEN_REGION-asf001.sp2.ygridcore.net,49518,1365336962323-1] regionserver.HRegion(496): Instantiated testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,g,1365336962398.682475e1358b7306bd3406e9330263a4. 2013-04-07 12:16:03,070 DEBUG [RS_OPEN_REGION-asf001.sp2.ygridcore.net,49518,1365336962323-2] zookeeper.ZKAssign(784): regionserver:49518-0x13de46b5f8b0012 Successfully transitioned node b1b340dff1e06fd6cb0b0a9c64964551 from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING 2013-04-07 12:16:03,070 DEBUG [RS_OPEN_REGION-asf001.sp2.ygridcore.net,60884,1365336878389-1] regionserver.MetricsRegionSourceImpl(54): Creating new MetricsRegionSourceImpl for table testRegionAssignmentAfterMasterRecoveryDueToZKExpiry eff8376859f5e6ba3201ec0ec9536d0f 2013-04-07 12:16:03,070 INFO [RS_OPEN_REGION-asf001.sp2.ygridcore.net,49518,1365336962323-2] regionserver.HRegion(4023): HRegion.openHRegion Region name ==testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,a,1365336962398.b1b340dff1e06fd6cb0b0a9c64964551. 2013-04-07 12:16:03,071 INFO [RS_OPEN_REGION-asf001.sp2.ygridcore.net,60884,1365336878389-2] regionserver.HRegion(4023): HRegion.openHRegion Region name ==testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,i,1365336962398.385e5846630b0cb4493758961a2c2457. 2013-04-07 12:16:03,071 DEBUG [RS_OPEN_REGION-asf001.sp2.ygridcore.net,60884,1365336878389-2] regionserver.HRegion(4025): Opening region: {NAME => 'testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,i,1365336962398.385e5846630b0cb4493758961a2c2457.', STARTKEY => 'i', ENDKEY => 'j', ENCODED => 385e5846630b0cb4493758961a2c2457,} 2013-04-07 12:16:03,071 DEBUG [RS_OPEN_REGION-asf001.sp2.ygridcore.net,49518,1365336962323-2] regionserver.HRegion(4025): Opening region: {NAME => 'testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,a,1365336962398.b1b340dff1e06fd6cb0b0a9c64964551.', STARTKEY => 'a', ENDKEY => 'b', ENCODED => b1b340dff1e06fd6cb0b0a9c64964551,} 2013-04-07 12:16:03,076 DEBUG [RS_OPEN_REGION-asf001.sp2.ygridcore.net,49518,1365336962323-2] regionserver.MetricsRegionSourceImpl(54): Creating new MetricsRegionSourceImpl for table testRegionAssignmentAfterMasterRecoveryDueToZKExpiry b1b340dff1e06fd6cb0b0a9c64964551 2013-04-07 12:16:03,077 DEBUG [RS_OPEN_REGION-asf001.sp2.ygridcore.net,49518,1365336962323-2] regionserver.HRegion(496): Instantiated testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,a,1365336962398.b1b340dff1e06fd6cb0b0a9c64964551. 2013-04-07 12:16:03,082 INFO [RS_OPEN_REGION-asf001.sp2.ygridcore.net,49518,1365336962323-0] regionserver.HRegion(586): Onlined testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,d,1365336962398.e99798d2c4d566ddb063efe03842457f.; next sequenceid=0 2013-04-07 12:16:03,082 INFO [RS_OPEN_REGION-asf001.sp2.ygridcore.net,49518,1365336962323-1] regionserver.HRegion(586): Onlined testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,g,1365336962398.682475e1358b7306bd3406e9330263a4.; next sequenceid=0 2013-04-07 12:16:03,082 DEBUG [RS_OPEN_REGION-asf001.sp2.ygridcore.net,49518,1365336962323-0] zookeeper.ZKAssign(721): regionserver:49518-0x13de46b5f8b0012 Attempting to transition node e99798d2c4d566ddb063efe03842457f from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENING 2013-04-07 12:16:03,082 DEBUG [RS_OPEN_REGION-asf001.sp2.ygridcore.net,49518,1365336962323-1] zookeeper.ZKAssign(721): regionserver:49518-0x13de46b5f8b0012 Attempting to transition node 682475e1358b7306bd3406e9330263a4 from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENING 2013-04-07 12:16:03,094 DEBUG [RS_OPEN_REGION-asf001.sp2.ygridcore.net,49518,1365336962323-0] zookeeper.ZKUtil(1680): regionserver:49518-0x13de46b5f8b0012 Retrieved 156 byte(s) of data from znode /hbase/region-in-transition/e99798d2c4d566ddb063efe03842457f; data=region=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,d,1365336962398.e99798d2c4d566ddb063efe03842457f., state=RS_ZK_REGION_OPENING, servername=asf001.sp2.ygridcore.net,49518,1365336962323, createTime=1365336963053, payload.length=0 2013-04-07 12:16:03,094 DEBUG [RS_OPEN_REGION-asf001.sp2.ygridcore.net,60884,1365336878389-1] zookeeper.ZKUtil(1680): regionserver:60884-0x13de46b5f8b0001 Retrieved 156 byte(s) of data from znode /hbase/region-in-transition/eff8376859f5e6ba3201ec0ec9536d0f; data=region=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,f,1365336962398.eff8376859f5e6ba3201ec0ec9536d0f., state=RS_ZK_REGION_OPENING, servername=asf001.sp2.ygridcore.net,60884,1365336878389, createTime=1365336963056, payload.length=0 2013-04-07 12:16:03,096 DEBUG [RS_OPEN_REGION-asf001.sp2.ygridcore.net,49518,1365336962323-1] zookeeper.ZKUtil(1680): regionserver:49518-0x13de46b5f8b0012 Retrieved 156 byte(s) of data from znode /hbase/region-in-transition/682475e1358b7306bd3406e9330263a4; data=region=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,g,1365336962398.682475e1358b7306bd3406e9330263a4., state=RS_ZK_REGION_OPENING, servername=asf001.sp2.ygridcore.net,49518,1365336962323, createTime=1365336963053, payload.length=0 2013-04-07 12:16:03,097 DEBUG [RS_OPEN_REGION-asf001.sp2.ygridcore.net,49518,1365336962323-0] zookeeper.ZKAssign(784): regionserver:49518-0x13de46b5f8b0012 Successfully transitioned node e99798d2c4d566ddb063efe03842457f from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENING 2013-04-07 12:16:03,098 INFO [PostOpenDeployTasks:e99798d2c4d566ddb063efe03842457f] regionserver.HRegionServer(1563): Post open deploy tasks for region=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,d,1365336962398.e99798d2c4d566ddb063efe03842457f. 2013-04-07 12:16:03,099 DEBUG [PostOpenDeployTasks:dbb82f7f432d6664327da0b8f15b7d76] zookeeper.ZKUtil(1680): hconnection-0x43ad4a-0x13de46b5f8b0005 Retrieved 43 byte(s) of data from znode /hbase/meta-region-server; data=asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:16:03,101 DEBUG [RS_OPEN_REGION-asf001.sp2.ygridcore.net,49518,1365336962323-1] zookeeper.ZKAssign(784): regionserver:49518-0x13de46b5f8b0012 Successfully transitioned node 682475e1358b7306bd3406e9330263a4 from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENING 2013-04-07 12:16:03,102 INFO [PostOpenDeployTasks:682475e1358b7306bd3406e9330263a4] regionserver.HRegionServer(1563): Post open deploy tasks for region=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,g,1365336962398.682475e1358b7306bd3406e9330263a4. 2013-04-07 12:16:03,109 DEBUG [PostOpenDeployTasks:e99798d2c4d566ddb063efe03842457f] zookeeper.ZKUtil(1680): hconnection-0x1f357d3-0x13de46b5f8b0013 Retrieved 43 byte(s) of data from znode /hbase/meta-region-server; data=asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:16:03,109 DEBUG [MASTER_TABLE_OPERATIONS-asf001.sp2.ygridcore.net,33796,1365336878036-0] master.TableLockManager$ZKTableLockManager$TableLockImpl(253): Attempt to release table write lock on :testRegionAssignmentAfterMasterRecoveryDueToZKExpiry 2013-04-07 12:16:03,116 DEBUG [PostOpenDeployTasks:682475e1358b7306bd3406e9330263a4] zookeeper.ZKUtil(1680): hconnection-0x1f357d3-0x13de46b5f8b0013 Retrieved 43 byte(s) of data from znode /hbase/meta-region-server; data=asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:16:03,117 DEBUG [PostOpenDeployTasks:e99798d2c4d566ddb063efe03842457f] zookeeper.ZKUtil(1680): hconnection-0x1f357d3-0x13de46b5f8b0013 Retrieved 43 byte(s) of data from znode /hbase/meta-region-server; data=asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:16:03,119 DEBUG [PostOpenDeployTasks:682475e1358b7306bd3406e9330263a4] zookeeper.ZKUtil(1680): hconnection-0x1f357d3-0x13de46b5f8b0013 Retrieved 43 byte(s) of data from znode /hbase/meta-region-server; data=asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:16:03,128 INFO [RS_OPEN_REGION-asf001.sp2.ygridcore.net,49518,1365336962323-2] regionserver.HRegion(586): Onlined testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,a,1365336962398.b1b340dff1e06fd6cb0b0a9c64964551.; next sequenceid=0 2013-04-07 12:16:03,128 DEBUG [RS_OPEN_REGION-asf001.sp2.ygridcore.net,49518,1365336962323-2] zookeeper.ZKAssign(721): regionserver:49518-0x13de46b5f8b0012 Attempting to transition node b1b340dff1e06fd6cb0b0a9c64964551 from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENING 2013-04-07 12:16:03,133 DEBUG [RS_OPEN_REGION-asf001.sp2.ygridcore.net,49518,1365336962323-2] zookeeper.ZKUtil(1680): regionserver:49518-0x13de46b5f8b0012 Retrieved 156 byte(s) of data from znode /hbase/region-in-transition/b1b340dff1e06fd6cb0b0a9c64964551; data=region=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,a,1365336962398.b1b340dff1e06fd6cb0b0a9c64964551., state=RS_ZK_REGION_OPENING, servername=asf001.sp2.ygridcore.net,49518,1365336962323, createTime=1365336963056, payload.length=0 2013-04-07 12:16:03,134 INFO [PostOpenDeployTasks:682475e1358b7306bd3406e9330263a4] catalog.MetaEditor(432): Updated row testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,g,1365336962398.682475e1358b7306bd3406e9330263a4. with server=asf001.sp2.ygridcore.net,49518,1365336962323 2013-04-07 12:16:03,134 INFO [PostOpenDeployTasks:682475e1358b7306bd3406e9330263a4] regionserver.HRegionServer(1584): Done with post open deploy task for region=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,g,1365336962398.682475e1358b7306bd3406e9330263a4. 2013-04-07 12:16:03,134 INFO [PostOpenDeployTasks:e99798d2c4d566ddb063efe03842457f] catalog.MetaEditor(432): Updated row testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,d,1365336962398.e99798d2c4d566ddb063efe03842457f. with server=asf001.sp2.ygridcore.net,49518,1365336962323 2013-04-07 12:16:03,134 DEBUG [RS_OPEN_REGION-asf001.sp2.ygridcore.net,49518,1365336962323-1] zookeeper.ZKAssign(721): regionserver:49518-0x13de46b5f8b0012 Attempting to transition node 682475e1358b7306bd3406e9330263a4 from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENED 2013-04-07 12:16:03,134 DEBUG [RS_OPEN_REGION-asf001.sp2.ygridcore.net,60884,1365336878389-0] zookeeper.ZKAssign(721): regionserver:60884-0x13de46b5f8b0001 Attempting to transition node dbb82f7f432d6664327da0b8f15b7d76 from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENED 2013-04-07 12:16:03,134 INFO [PostOpenDeployTasks:e99798d2c4d566ddb063efe03842457f] regionserver.HRegionServer(1584): Done with post open deploy task for region=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,d,1365336962398.e99798d2c4d566ddb063efe03842457f. 2013-04-07 12:16:03,134 DEBUG [RS_OPEN_REGION-asf001.sp2.ygridcore.net,49518,1365336962323-0] zookeeper.ZKAssign(721): regionserver:49518-0x13de46b5f8b0012 Attempting to transition node e99798d2c4d566ddb063efe03842457f from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENED 2013-04-07 12:16:03,142 DEBUG [RS_OPEN_REGION-asf001.sp2.ygridcore.net,49518,1365336962323-2] zookeeper.ZKAssign(784): regionserver:49518-0x13de46b5f8b0012 Successfully transitioned node b1b340dff1e06fd6cb0b0a9c64964551 from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENING 2013-04-07 12:16:03,142 INFO [PostOpenDeployTasks:b1b340dff1e06fd6cb0b0a9c64964551] regionserver.HRegionServer(1563): Post open deploy tasks for region=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,a,1365336962398.b1b340dff1e06fd6cb0b0a9c64964551. 2013-04-07 12:16:03,146 DEBUG [PostOpenDeployTasks:b1b340dff1e06fd6cb0b0a9c64964551] zookeeper.ZKUtil(1680): hconnection-0x1f357d3-0x13de46b5f8b0013 Retrieved 43 byte(s) of data from znode /hbase/meta-region-server; data=asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:16:03,154 DEBUG [RS_OPEN_REGION-asf001.sp2.ygridcore.net,49518,1365336962323-1] zookeeper.ZKUtil(1680): regionserver:49518-0x13de46b5f8b0012 Retrieved 156 byte(s) of data from znode /hbase/region-in-transition/682475e1358b7306bd3406e9330263a4; data=region=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,g,1365336962398.682475e1358b7306bd3406e9330263a4., state=RS_ZK_REGION_OPENING, servername=asf001.sp2.ygridcore.net,49518,1365336962323, createTime=1365336963096, payload.length=0 2013-04-07 12:16:03,154 DEBUG [PostOpenDeployTasks:385e5846630b0cb4493758961a2c2457] zookeeper.ZKUtil(1680): hconnection-0x43ad4a-0x13de46b5f8b0005 Retrieved 43 byte(s) of data from znode /hbase/meta-region-server; data=asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:16:03,154 DEBUG [RS_OPEN_REGION-asf001.sp2.ygridcore.net,60884,1365336878389-1] zookeeper.ZKUtil(1680): regionserver:60884-0x13de46b5f8b0001 Retrieved 156 byte(s) of data from znode /hbase/region-in-transition/eff8376859f5e6ba3201ec0ec9536d0f; data=region=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,f,1365336962398.eff8376859f5e6ba3201ec0ec9536d0f., state=RS_ZK_REGION_OPENING, servername=asf001.sp2.ygridcore.net,60884,1365336878389, createTime=1365336963094, payload.length=0 2013-04-07 12:16:03,155 DEBUG [RS_OPEN_REGION-asf001.sp2.ygridcore.net,60884,1365336878389-0] zookeeper.ZKUtil(1680): regionserver:60884-0x13de46b5f8b0001 Retrieved 156 byte(s) of data from znode /hbase/region-in-transition/dbb82f7f432d6664327da0b8f15b7d76; data=region=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,e,1365336962398.dbb82f7f432d6664327da0b8f15b7d76., state=RS_ZK_REGION_OPENING, servername=asf001.sp2.ygridcore.net,60884,1365336878389, createTime=1365336963084, payload.length=0 2013-04-07 12:16:03,156 DEBUG [PostOpenDeployTasks:b1b340dff1e06fd6cb0b0a9c64964551] zookeeper.ZKUtil(1680): hconnection-0x1f357d3-0x13de46b5f8b0013 Retrieved 43 byte(s) of data from znode /hbase/meta-region-server; data=asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:16:03,160 INFO [PostOpenDeployTasks:b1b340dff1e06fd6cb0b0a9c64964551] catalog.MetaEditor(432): Updated row testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,a,1365336962398.b1b340dff1e06fd6cb0b0a9c64964551. with server=asf001.sp2.ygridcore.net,49518,1365336962323 2013-04-07 12:16:03,160 INFO [PostOpenDeployTasks:b1b340dff1e06fd6cb0b0a9c64964551] regionserver.HRegionServer(1584): Done with post open deploy task for region=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,a,1365336962398.b1b340dff1e06fd6cb0b0a9c64964551. 2013-04-07 12:16:03,162 DEBUG [RS_OPEN_REGION-asf001.sp2.ygridcore.net,49518,1365336962323-2] zookeeper.ZKAssign(721): regionserver:49518-0x13de46b5f8b0012 Attempting to transition node b1b340dff1e06fd6cb0b0a9c64964551 from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENED 2013-04-07 12:16:03,163 DEBUG [RS_OPEN_REGION-asf001.sp2.ygridcore.net,49518,1365336962323-0] zookeeper.ZKUtil(1680): regionserver:49518-0x13de46b5f8b0012 Retrieved 156 byte(s) of data from znode /hbase/region-in-transition/e99798d2c4d566ddb063efe03842457f; data=region=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,d,1365336962398.e99798d2c4d566ddb063efe03842457f., state=RS_ZK_REGION_OPENING, servername=asf001.sp2.ygridcore.net,49518,1365336962323, createTime=1365336963094, payload.length=0 2013-04-07 12:16:03,163 DEBUG [RS_OPEN_REGION-asf001.sp2.ygridcore.net,60884,1365336878389-1] zookeeper.ZKAssign(784): regionserver:60884-0x13de46b5f8b0001 Successfully transitioned node eff8376859f5e6ba3201ec0ec9536d0f from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENED 2013-04-07 12:16:03,163 DEBUG [RS_OPEN_REGION-asf001.sp2.ygridcore.net,60884,1365336878389-1] handler.OpenRegionHandler(361): region transitioned to opened in zookeeper: {NAME => 'testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,f,1365336962398.eff8376859f5e6ba3201ec0ec9536d0f.', STARTKEY => 'f', ENDKEY => 'g', ENCODED => eff8376859f5e6ba3201ec0ec9536d0f,}, server: asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:16:03,165 DEBUG [RS_OPEN_REGION-asf001.sp2.ygridcore.net,49518,1365336962323-1] zookeeper.ZKAssign(784): regionserver:49518-0x13de46b5f8b0012 Successfully transitioned node 682475e1358b7306bd3406e9330263a4 from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENED 2013-04-07 12:16:03,165 DEBUG [RS_OPEN_REGION-asf001.sp2.ygridcore.net,49518,1365336962323-1] handler.OpenRegionHandler(361): region transitioned to opened in zookeeper: {NAME => 'testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,g,1365336962398.682475e1358b7306bd3406e9330263a4.', STARTKEY => 'g', ENDKEY => 'h', ENCODED => 682475e1358b7306bd3406e9330263a4,}, server: asf001.sp2.ygridcore.net,49518,1365336962323 2013-04-07 12:16:03,165 DEBUG [RS_OPEN_REGION-asf001.sp2.ygridcore.net,49518,1365336962323-1] handler.OpenRegionHandler(168): Opened testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,g,1365336962398.682475e1358b7306bd3406e9330263a4. on server:asf001.sp2.ygridcore.net,49518,1365336962323 2013-04-07 12:16:03,165 DEBUG [RS_OPEN_REGION-asf001.sp2.ygridcore.net,49518,1365336962323-1] zookeeper.ZKAssign(721): regionserver:49518-0x13de46b5f8b0012 Attempting to transition node 38f0eb146fb473b19541deaada3419b9 from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING 2013-04-07 12:16:03,166 DEBUG [RS_OPEN_REGION-asf001.sp2.ygridcore.net,60884,1365336878389-0] zookeeper.ZKAssign(784): regionserver:60884-0x13de46b5f8b0001 Successfully transitioned node dbb82f7f432d6664327da0b8f15b7d76 from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENED 2013-04-07 12:16:03,166 DEBUG [RS_OPEN_REGION-asf001.sp2.ygridcore.net,60884,1365336878389-0] handler.OpenRegionHandler(361): region transitioned to opened in zookeeper: {NAME => 'testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,e,1365336962398.dbb82f7f432d6664327da0b8f15b7d76.', STARTKEY => 'e', ENDKEY => 'f', ENCODED => dbb82f7f432d6664327da0b8f15b7d76,}, server: asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:16:03,166 DEBUG [RS_OPEN_REGION-asf001.sp2.ygridcore.net,60884,1365336878389-0] handler.OpenRegionHandler(168): Opened testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,e,1365336962398.dbb82f7f432d6664327da0b8f15b7d76. on server:asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:16:03,169 DEBUG [RS_OPEN_REGION-asf001.sp2.ygridcore.net,49518,1365336962323-2] zookeeper.ZKUtil(1680): regionserver:49518-0x13de46b5f8b0012 Retrieved 156 byte(s) of data from znode /hbase/region-in-transition/b1b340dff1e06fd6cb0b0a9c64964551; data=region=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,a,1365336962398.b1b340dff1e06fd6cb0b0a9c64964551., state=RS_ZK_REGION_OPENING, servername=asf001.sp2.ygridcore.net,49518,1365336962323, createTime=1365336963133, payload.length=0 2013-04-07 12:16:03,172 DEBUG [RS_OPEN_REGION-asf001.sp2.ygridcore.net,49518,1365336962323-0] zookeeper.ZKAssign(784): regionserver:49518-0x13de46b5f8b0012 Successfully transitioned node e99798d2c4d566ddb063efe03842457f from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENED 2013-04-07 12:16:03,172 DEBUG [RS_OPEN_REGION-asf001.sp2.ygridcore.net,49518,1365336962323-0] handler.OpenRegionHandler(361): region transitioned to opened in zookeeper: {NAME => 'testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,d,1365336962398.e99798d2c4d566ddb063efe03842457f.', STARTKEY => 'd', ENDKEY => 'e', ENCODED => e99798d2c4d566ddb063efe03842457f,}, server: asf001.sp2.ygridcore.net,49518,1365336962323 2013-04-07 12:16:03,172 DEBUG [RS_OPEN_REGION-asf001.sp2.ygridcore.net,49518,1365336962323-0] handler.OpenRegionHandler(168): Opened testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,d,1365336962398.e99798d2c4d566ddb063efe03842457f. on server:asf001.sp2.ygridcore.net,49518,1365336962323 2013-04-07 12:16:03,172 DEBUG [RS_OPEN_REGION-asf001.sp2.ygridcore.net,49518,1365336962323-1] zookeeper.ZKUtil(1680): regionserver:49518-0x13de46b5f8b0012 Retrieved 156 byte(s) of data from znode /hbase/region-in-transition/38f0eb146fb473b19541deaada3419b9; data=region=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,j,1365336962398.38f0eb146fb473b19541deaada3419b9., state=M_ZK_REGION_OFFLINE, servername=asf001.sp2.ygridcore.net,49518,1365336962323, createTime=1365336962997, payload.length=0 2013-04-07 12:16:03,173 DEBUG [RS_OPEN_REGION-asf001.sp2.ygridcore.net,60884,1365336878389-1] zookeeper.ZKAssign(784): regionserver:60884-0x13de46b5f8b0001 Successfully transitioned node 11c8123e8d563f264d947ffa106e4c7c from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING 2013-04-07 12:16:03,173 INFO [RS_OPEN_REGION-asf001.sp2.ygridcore.net,60884,1365336878389-1] regionserver.HRegion(4023): HRegion.openHRegion Region name ==testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,b,1365336962398.11c8123e8d563f264d947ffa106e4c7c. 2013-04-07 12:16:03,174 DEBUG [RS_OPEN_REGION-asf001.sp2.ygridcore.net,60884,1365336878389-1] regionserver.HRegion(4025): Opening region: {NAME => 'testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,b,1365336962398.11c8123e8d563f264d947ffa106e4c7c.', STARTKEY => 'b', ENDKEY => 'c', ENCODED => 11c8123e8d563f264d947ffa106e4c7c,} 2013-04-07 12:16:03,175 DEBUG [RS_OPEN_REGION-asf001.sp2.ygridcore.net,60884,1365336878389-2] zookeeper.ZKAssign(784): regionserver:60884-0x13de46b5f8b0001 Successfully transitioned node 385e5846630b0cb4493758961a2c2457 from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENED 2013-04-07 12:16:03,175 DEBUG [RS_OPEN_REGION-asf001.sp2.ygridcore.net,60884,1365336878389-2] handler.OpenRegionHandler(361): region transitioned to opened in zookeeper: {NAME => 'testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,i,1365336962398.385e5846630b0cb4493758961a2c2457.', STARTKEY => 'i', ENDKEY => 'j', ENCODED => 385e5846630b0cb4493758961a2c2457,}, server: asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:16:03,175 DEBUG [RS_OPEN_REGION-asf001.sp2.ygridcore.net,60884,1365336878389-2] handler.OpenRegionHandler(168): Opened testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,i,1365336962398.385e5846630b0cb4493758961a2c2457. on server:asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:16:03,175 DEBUG [RS_OPEN_REGION-asf001.sp2.ygridcore.net,49518,1365336962323-2] zookeeper.ZKAssign(784): regionserver:49518-0x13de46b5f8b0012 Successfully transitioned node b1b340dff1e06fd6cb0b0a9c64964551 from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENED 2013-04-07 12:16:03,175 DEBUG [RS_OPEN_REGION-asf001.sp2.ygridcore.net,49518,1365336962323-2] handler.OpenRegionHandler(361): region transitioned to opened in zookeeper: {NAME => 'testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,a,1365336962398.b1b340dff1e06fd6cb0b0a9c64964551.', STARTKEY => 'a', ENDKEY => 'b', ENCODED => b1b340dff1e06fd6cb0b0a9c64964551,}, server: asf001.sp2.ygridcore.net,49518,1365336962323 2013-04-07 12:16:03,175 DEBUG [RS_OPEN_REGION-asf001.sp2.ygridcore.net,49518,1365336962323-2] handler.OpenRegionHandler(168): Opened testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,a,1365336962398.b1b340dff1e06fd6cb0b0a9c64964551. on server:asf001.sp2.ygridcore.net,49518,1365336962323 2013-04-07 12:16:03,175 DEBUG [RS_OPEN_REGION-asf001.sp2.ygridcore.net,60884,1365336878389-1] regionserver.MetricsRegionSourceImpl(54): Creating new MetricsRegionSourceImpl for table testRegionAssignmentAfterMasterRecoveryDueToZKExpiry 11c8123e8d563f264d947ffa106e4c7c 2013-04-07 12:16:03,176 DEBUG [RS_OPEN_REGION-asf001.sp2.ygridcore.net,49518,1365336962323-1] zookeeper.ZKAssign(784): regionserver:49518-0x13de46b5f8b0012 Successfully transitioned node 38f0eb146fb473b19541deaada3419b9 from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING 2013-04-07 12:16:03,176 INFO [RS_OPEN_REGION-asf001.sp2.ygridcore.net,49518,1365336962323-1] regionserver.HRegion(4023): HRegion.openHRegion Region name ==testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,j,1365336962398.38f0eb146fb473b19541deaada3419b9. 2013-04-07 12:16:03,176 DEBUG [RS_OPEN_REGION-asf001.sp2.ygridcore.net,49518,1365336962323-1] regionserver.HRegion(4025): Opening region: {NAME => 'testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,j,1365336962398.38f0eb146fb473b19541deaada3419b9.', STARTKEY => 'j', ENDKEY => '', ENCODED => 38f0eb146fb473b19541deaada3419b9,} 2013-04-07 12:16:03,178 DEBUG [RS_OPEN_REGION-asf001.sp2.ygridcore.net,49518,1365336962323-1] regionserver.MetricsRegionSourceImpl(54): Creating new MetricsRegionSourceImpl for table testRegionAssignmentAfterMasterRecoveryDueToZKExpiry 38f0eb146fb473b19541deaada3419b9 2013-04-07 12:16:03,178 DEBUG [RS_OPEN_REGION-asf001.sp2.ygridcore.net,49518,1365336962323-1] regionserver.HRegion(496): Instantiated testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,j,1365336962398.38f0eb146fb473b19541deaada3419b9. 2013-04-07 12:16:03,184 INFO [RS_OPEN_REGION-asf001.sp2.ygridcore.net,49518,1365336962323-1] regionserver.HRegion(586): Onlined testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,j,1365336962398.38f0eb146fb473b19541deaada3419b9.; next sequenceid=0 2013-04-07 12:16:03,185 DEBUG [RS_OPEN_REGION-asf001.sp2.ygridcore.net,49518,1365336962323-1] zookeeper.ZKAssign(721): regionserver:49518-0x13de46b5f8b0012 Attempting to transition node 38f0eb146fb473b19541deaada3419b9 from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENING 2013-04-07 12:16:03,185 DEBUG [MASTER_OPEN_REGION-asf001.sp2.ygridcore.net,33796,1365336878036-0] handler.OpenedRegionHandler(146): Handling OPENED event for testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,,1365336962397.d4e129e51e67d798e17083623fdde133. from asf001.sp2.ygridcore.net,43802,1365336889418; deleting unassigned node 2013-04-07 12:16:03,186 DEBUG [MASTER_OPEN_REGION-asf001.sp2.ygridcore.net,33796,1365336878036-0] zookeeper.ZKAssign(405): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Deleting existing unassigned node for d4e129e51e67d798e17083623fdde133 that is in expected state RS_ZK_REGION_OPENED 2013-04-07 12:16:03,197 DEBUG [RS_OPEN_REGION-asf001.sp2.ygridcore.net,49518,1365336962323-1] zookeeper.ZKUtil(1680): regionserver:49518-0x13de46b5f8b0012 Retrieved 156 byte(s) of data from znode /hbase/region-in-transition/38f0eb146fb473b19541deaada3419b9; data=region=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,j,1365336962398.38f0eb146fb473b19541deaada3419b9., state=RS_ZK_REGION_OPENING, servername=asf001.sp2.ygridcore.net,49518,1365336962323, createTime=1365336963173, payload.length=0 2013-04-07 12:16:03,197 DEBUG [RS_OPEN_REGION-asf001.sp2.ygridcore.net,60884,1365336878389-1] zookeeper.ZKAssign(784): regionserver:60884-0x13de46b5f8b0001 Successfully transitioned node 11c8123e8d563f264d947ffa106e4c7c from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENING 2013-04-07 12:16:03,200 DEBUG [RS_OPEN_REGION-asf001.sp2.ygridcore.net,49518,1365336962323-1] zookeeper.ZKAssign(784): regionserver:49518-0x13de46b5f8b0012 Successfully transitioned node 38f0eb146fb473b19541deaada3419b9 from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENING 2013-04-07 12:16:03,202 INFO [PostOpenDeployTasks:38f0eb146fb473b19541deaada3419b9] regionserver.HRegionServer(1563): Post open deploy tasks for region=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,j,1365336962398.38f0eb146fb473b19541deaada3419b9. 2013-04-07 12:16:03,202 DEBUG [PostOpenDeployTasks:11c8123e8d563f264d947ffa106e4c7c] zookeeper.ZKUtil(1680): hconnection-0x43ad4a-0x13de46b5f8b0005 Retrieved 43 byte(s) of data from znode /hbase/meta-region-server; data=asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:16:03,204 DEBUG [PostOpenDeployTasks:38f0eb146fb473b19541deaada3419b9] zookeeper.ZKUtil(1680): hconnection-0x1f357d3-0x13de46b5f8b0013 Retrieved 43 byte(s) of data from znode /hbase/meta-region-server; data=asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:16:03,207 DEBUG [PostOpenDeployTasks:38f0eb146fb473b19541deaada3419b9] zookeeper.ZKUtil(1680): hconnection-0x1f357d3-0x13de46b5f8b0013 Retrieved 43 byte(s) of data from znode /hbase/meta-region-server; data=asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:16:03,215 INFO [PostOpenDeployTasks:38f0eb146fb473b19541deaada3419b9] catalog.MetaEditor(432): Updated row testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,j,1365336962398.38f0eb146fb473b19541deaada3419b9. with server=asf001.sp2.ygridcore.net,49518,1365336962323 2013-04-07 12:16:03,215 DEBUG [RS_OPEN_REGION-asf001.sp2.ygridcore.net,60884,1365336878389-1] zookeeper.ZKAssign(784): regionserver:60884-0x13de46b5f8b0001 Successfully transitioned node 11c8123e8d563f264d947ffa106e4c7c from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENED 2013-04-07 12:16:03,216 INFO [PostOpenDeployTasks:38f0eb146fb473b19541deaada3419b9] regionserver.HRegionServer(1584): Done with post open deploy task for region=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,j,1365336962398.38f0eb146fb473b19541deaada3419b9. 2013-04-07 12:16:03,216 DEBUG [RS_OPEN_REGION-asf001.sp2.ygridcore.net,60884,1365336878389-1] handler.OpenRegionHandler(361): region transitioned to opened in zookeeper: {NAME => 'testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,b,1365336962398.11c8123e8d563f264d947ffa106e4c7c.', STARTKEY => 'b', ENDKEY => 'c', ENCODED => 11c8123e8d563f264d947ffa106e4c7c,}, server: asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:16:03,216 DEBUG [RS_OPEN_REGION-asf001.sp2.ygridcore.net,49518,1365336962323-1] zookeeper.ZKAssign(721): regionserver:49518-0x13de46b5f8b0012 Attempting to transition node 38f0eb146fb473b19541deaada3419b9 from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENED 2013-04-07 12:16:03,216 DEBUG [RS_OPEN_REGION-asf001.sp2.ygridcore.net,60884,1365336878389-1] handler.OpenRegionHandler(168): Opened testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,b,1365336962398.11c8123e8d563f264d947ffa106e4c7c. on server:asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:16:03,218 DEBUG [RS_OPEN_REGION-asf001.sp2.ygridcore.net,49518,1365336962323-1] zookeeper.ZKUtil(1680): regionserver:49518-0x13de46b5f8b0012 Retrieved 156 byte(s) of data from znode /hbase/region-in-transition/38f0eb146fb473b19541deaada3419b9; data=region=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,j,1365336962398.38f0eb146fb473b19541deaada3419b9., state=RS_ZK_REGION_OPENING, servername=asf001.sp2.ygridcore.net,49518,1365336962323, createTime=1365336963197, payload.length=0 2013-04-07 12:16:03,221 DEBUG [RS_OPEN_REGION-asf001.sp2.ygridcore.net,49518,1365336962323-1] zookeeper.ZKAssign(784): regionserver:49518-0x13de46b5f8b0012 Successfully transitioned node 38f0eb146fb473b19541deaada3419b9 from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENED 2013-04-07 12:16:03,221 DEBUG [RS_OPEN_REGION-asf001.sp2.ygridcore.net,49518,1365336962323-1] handler.OpenRegionHandler(361): region transitioned to opened in zookeeper: {NAME => 'testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,j,1365336962398.38f0eb146fb473b19541deaada3419b9.', STARTKEY => 'j', ENDKEY => '', ENCODED => 38f0eb146fb473b19541deaada3419b9,}, server: asf001.sp2.ygridcore.net,49518,1365336962323 2013-04-07 12:16:03,221 DEBUG [RS_OPEN_REGION-asf001.sp2.ygridcore.net,49518,1365336962323-1] handler.OpenRegionHandler(168): Opened testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,j,1365336962398.38f0eb146fb473b19541deaada3419b9. on server:asf001.sp2.ygridcore.net,49518,1365336962323 2013-04-07 12:16:03,222 DEBUG [MASTER_OPEN_REGION-asf001.sp2.ygridcore.net,33796,1365336878036-2] zookeeper.ZKUtil(1680): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Retrieved 156 byte(s) of data from znode /hbase/region-in-transition/eff8376859f5e6ba3201ec0ec9536d0f; data=region=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,f,1365336962398.eff8376859f5e6ba3201ec0ec9536d0f., state=RS_ZK_REGION_OPENED, servername=asf001.sp2.ygridcore.net,60884,1365336878389, createTime=1365336963154, payload.length=0 2013-04-07 12:16:03,234 INFO [PRI IPC Server handler 1 on 49518] regionserver.HRegionServer(3434): Received close region: b1b340dff1e06fd6cb0b0a9c64964551Transitioning in ZK: no. Version of ZK closing node:-1. Destination server:null 2013-04-07 12:16:03,235 DEBUG [RS_CLOSE_REGION-asf001.sp2.ygridcore.net,49518,1365336962323-0] handler.CloseRegionHandler(125): Processing close of testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,a,1365336962398.b1b340dff1e06fd6cb0b0a9c64964551. 2013-04-07 12:16:03,236 DEBUG [RS_CLOSE_REGION-asf001.sp2.ygridcore.net,49518,1365336962323-0] regionserver.HRegion(889): Closing testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,a,1365336962398.b1b340dff1e06fd6cb0b0a9c64964551.: disabling compactions & flushes 2013-04-07 12:16:03,236 DEBUG [RS_CLOSE_REGION-asf001.sp2.ygridcore.net,49518,1365336962323-0] regionserver.HRegion(911): Updates disabled for region testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,a,1365336962398.b1b340dff1e06fd6cb0b0a9c64964551. 2013-04-07 12:16:03,236 INFO [RS_CLOSE_REGION-asf001.sp2.ygridcore.net,49518,1365336962323-0] regionserver.HRegion(968): Closed testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,a,1365336962398.b1b340dff1e06fd6cb0b0a9c64964551. 2013-04-07 12:16:03,237 DEBUG [RS_CLOSE_REGION-asf001.sp2.ygridcore.net,49518,1365336962323-0] handler.CloseRegionHandler(177): Closed region testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,a,1365336962398.b1b340dff1e06fd6cb0b0a9c64964551. 2013-04-07 12:16:03,254 INFO [PRI IPC Server handler 2 on 49518] regionserver.HRegionServer(3434): Received close region: 38f0eb146fb473b19541deaada3419b9Transitioning in ZK: no. Version of ZK closing node:-1. Destination server:null 2013-04-07 12:16:03,254 DEBUG [RS_CLOSE_REGION-asf001.sp2.ygridcore.net,49518,1365336962323-1] handler.CloseRegionHandler(125): Processing close of testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,j,1365336962398.38f0eb146fb473b19541deaada3419b9. 2013-04-07 12:16:03,254 INFO [PRI IPC Server handler 3 on 49518] regionserver.HRegionServer(3434): Received close region: 38f0eb146fb473b19541deaada3419b9Transitioning in ZK: no. Version of ZK closing node:-1. Destination server:null 2013-04-07 12:16:03,255 INFO [PRI IPC Server handler 3 on 49518] regionserver.HRegionServer(2334): Received CLOSE for the region: 38f0eb146fb473b19541deaada3419b9 ,which we are already trying to CLOSE 2013-04-07 12:16:03,259 DEBUG [RS_CLOSE_REGION-asf001.sp2.ygridcore.net,49518,1365336962323-1] regionserver.HRegion(889): Closing testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,j,1365336962398.38f0eb146fb473b19541deaada3419b9.: disabling compactions & flushes 2013-04-07 12:16:03,259 DEBUG [RS_CLOSE_REGION-asf001.sp2.ygridcore.net,49518,1365336962323-1] regionserver.HRegion(911): Updates disabled for region testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,j,1365336962398.38f0eb146fb473b19541deaada3419b9. 2013-04-07 12:16:03,259 INFO [RS_CLOSE_REGION-asf001.sp2.ygridcore.net,49518,1365336962323-1] regionserver.HRegion(968): Closed testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,j,1365336962398.38f0eb146fb473b19541deaada3419b9. 2013-04-07 12:16:03,259 DEBUG [RS_CLOSE_REGION-asf001.sp2.ygridcore.net,49518,1365336962323-1] handler.CloseRegionHandler(177): Closed region testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,j,1365336962398.38f0eb146fb473b19541deaada3419b9. 2013-04-07 12:16:03,260 WARN [MASTER_OPEN_REGION-asf001.sp2.ygridcore.net,33796,1365336878036-2] zookeeper.RecoverableZooKeeper(161): Node /hbase/region-in-transition/e99798d2c4d566ddb063efe03842457f already deleted, retry=false 2013-04-07 12:16:03,260 WARN [MASTER_OPEN_REGION-asf001.sp2.ygridcore.net,33796,1365336878036-2] handler.OpenedRegionHandler(132): The znode of the region testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,d,1365336962398.e99798d2c4d566ddb063efe03842457f. would have already been deleted 2013-04-07 12:16:03,260 ERROR [MASTER_OPEN_REGION-asf001.sp2.ygridcore.net,33796,1365336878036-2] handler.OpenedRegionHandler(104): The znode of region testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,d,1365336962398.e99798d2c4d566ddb063efe03842457f. could not be deleted. 2013-04-07 12:16:03,260 DEBUG [MASTER_OPEN_REGION-asf001.sp2.ygridcore.net,33796,1365336878036-2] handler.OpenedRegionHandler(146): Handling OPENED event for testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,i,1365336962398.385e5846630b0cb4493758961a2c2457. from asf001.sp2.ygridcore.net,60884,1365336878389; deleting unassigned node 2013-04-07 12:16:03,260 DEBUG [MASTER_OPEN_REGION-asf001.sp2.ygridcore.net,33796,1365336878036-2] zookeeper.ZKAssign(405): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Deleting existing unassigned node for 385e5846630b0cb4493758961a2c2457 that is in expected state RS_ZK_REGION_OPENED 2013-04-07 12:16:03,423 DEBUG [pool-1-thread-1] client.MetaScanner(198): Scanning .META. starting at row=testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,,00000000000000 for max=2147483647 rows using hconnection-0xfe04c6 2013-04-07 12:16:03,425 DEBUG [pool-1-thread-1] zookeeper.ZKUtil(1680): hconnection-0xfe04c6-0x13de46b5f8b0007 Retrieved 43 byte(s) of data from znode /hbase/meta-region-server; data=asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:16:03,431 DEBUG [pool-1-thread-1] hbase.HRegionInfo(954): Current INFO from scan results = {NAME => 'testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,,1365336962397.d4e129e51e67d798e17083623fdde133.', STARTKEY => '', ENDKEY => 'a', ENCODED => d4e129e51e67d798e17083623fdde133,} 2013-04-07 12:16:03,431 DEBUG [pool-1-thread-1] hbase.HRegionInfo(954): Current INFO from scan results = {NAME => 'testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,a,1365336962398.b1b340dff1e06fd6cb0b0a9c64964551.', STARTKEY => 'a', ENDKEY => 'b', ENCODED => b1b340dff1e06fd6cb0b0a9c64964551,} 2013-04-07 12:16:03,431 DEBUG [pool-1-thread-1] hbase.HRegionInfo(954): Current INFO from scan results = {NAME => 'testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,b,1365336962398.11c8123e8d563f264d947ffa106e4c7c.', STARTKEY => 'b', ENDKEY => 'c', ENCODED => 11c8123e8d563f264d947ffa106e4c7c,} 2013-04-07 12:16:03,432 DEBUG [pool-1-thread-1] hbase.HRegionInfo(954): Current INFO from scan results = {NAME => 'testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,c,1365336962398.4432e800611c00a4ae168862a273fc70.', STARTKEY => 'c', ENDKEY => 'd', ENCODED => 4432e800611c00a4ae168862a273fc70,} 2013-04-07 12:16:03,432 DEBUG [pool-1-thread-1] hbase.HRegionInfo(954): Current INFO from scan results = {NAME => 'testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,d,1365336962398.e99798d2c4d566ddb063efe03842457f.', STARTKEY => 'd', ENDKEY => 'e', ENCODED => e99798d2c4d566ddb063efe03842457f,} 2013-04-07 12:16:03,432 DEBUG [pool-1-thread-1] hbase.HRegionInfo(954): Current INFO from scan results = {NAME => 'testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,e,1365336962398.dbb82f7f432d6664327da0b8f15b7d76.', STARTKEY => 'e', ENDKEY => 'f', ENCODED => dbb82f7f432d6664327da0b8f15b7d76,} 2013-04-07 12:16:03,433 DEBUG [pool-1-thread-1] hbase.HRegionInfo(954): Current INFO from scan results = {NAME => 'testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,f,1365336962398.eff8376859f5e6ba3201ec0ec9536d0f.', STARTKEY => 'f', ENDKEY => 'g', ENCODED => eff8376859f5e6ba3201ec0ec9536d0f,} 2013-04-07 12:16:03,433 DEBUG [pool-1-thread-1] hbase.HRegionInfo(954): Current INFO from scan results = {NAME => 'testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,g,1365336962398.682475e1358b7306bd3406e9330263a4.', STARTKEY => 'g', ENDKEY => 'h', ENCODED => 682475e1358b7306bd3406e9330263a4,} 2013-04-07 12:16:03,433 DEBUG [pool-1-thread-1] hbase.HRegionInfo(954): Current INFO from scan results = {NAME => 'testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,h,1365336962398.1ce6d62427641e2a0e135d3d27b54b5d.', STARTKEY => 'h', ENDKEY => 'i', ENCODED => 1ce6d62427641e2a0e135d3d27b54b5d,} 2013-04-07 12:16:03,434 DEBUG [pool-1-thread-1] hbase.HRegionInfo(954): Current INFO from scan results = {NAME => 'testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,i,1365336962398.385e5846630b0cb4493758961a2c2457.', STARTKEY => 'i', ENDKEY => 'j', ENCODED => 385e5846630b0cb4493758961a2c2457,} 2013-04-07 12:16:03,434 DEBUG [pool-1-thread-1] hbase.HRegionInfo(954): Current INFO from scan results = {NAME => 'testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,j,1365336962398.38f0eb146fb473b19541deaada3419b9.', STARTKEY => 'j', ENDKEY => '', ENCODED => 38f0eb146fb473b19541deaada3419b9,} 2013-04-07 12:16:03,436 DEBUG [pool-1-thread-1] zookeeper.ZKUtil(1680): hconnection-0xfe04c6-0x13de46b5f8b0007 Retrieved 6 byte(s) of data from znode /hbase/table/testRegionAssignmentAfterMasterRecoveryDueToZKExpiry; data=PBUF\x08\x00 2013-04-07 12:16:03,436 DEBUG [pool-1-thread-1] zookeeper.ZKConfig(84): Skipped reading ZK properties file 'zoo.cfg' since 'hbase.config.read.zookeeper.config' was not set to true 2013-04-07 12:16:03,437 DEBUG [pool-1-thread-1] zookeeper.ZKUtil(119): unittest opening connection to ZooKeeper with ensemble (localhost:51597) 2013-04-07 12:16:03,438 INFO [pool-1-thread-1] zookeeper.RecoverableZooKeeper(119): The identifier of this process is unittest 2013-04-07 12:16:03,441 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(274): unittest Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2013-04-07 12:16:03,442 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(351): unittest-0x13de46b5f8b0014 connected 2013-04-07 12:16:03,442 DEBUG [RegionServer:3;asf001.sp2.ygridcore.net,49518,1365336962323-EventThread] zookeeper.ZooKeeperWatcher(274): regionserver:49518-0x13de46b5f8b0012 Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2013-04-07 12:16:03,443 FATAL [pool-1-thread-1] master.HMaster(1968): Master server abort: loaded coprocessors are: [org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint] 2013-04-07 12:16:03,443 INFO [pool-1-thread-1] master.HMaster(2052): Primary Master trying to recover from ZooKeeper session expiry. 2013-04-07 12:16:03,444 DEBUG [RegionServer:0;asf001.sp2.ygridcore.net,60884,1365336878389-EventThread] zookeeper.ZooKeeperWatcher(274): regionserver:60884-0x13de46b5f8b0001 Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2013-04-07 12:16:03,444 INFO [pool-1-thread-1] zookeeper.RecoverableZooKeeper(131): Closing dead ZooKeeper connection, session was: 0x13de46b5f8b000e 2013-04-07 12:16:03,444 INFO [pool-1-thread-1] zookeeper.RecoverableZooKeeper(136): Recreated a ZooKeeper, session is: 0x0 2013-04-07 12:16:03,447 DEBUG [RegionServer:0;asf001.sp2.ygridcore.net,60884,1365336878389-EventThread] zookeeper.ZKUtil(424): regionserver:60884-0x13de46b5f8b0001 /hbase/master does not exist. Watcher is set. 2013-04-07 12:16:03,447 DEBUG [RegionServer:3;asf001.sp2.ygridcore.net,49518,1365336962323-EventThread] zookeeper.ZKUtil(424): regionserver:49518-0x13de46b5f8b0012 /hbase/master does not exist. Watcher is set. 2013-04-07 12:16:03,448 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(274): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2013-04-07 12:16:03,449 DEBUG [pool-96-thread-1] zookeeper.ZKUtil(422): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e Set watcher on existing znode /hbase/running 2013-04-07 12:16:03,449 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(351): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e-0x13de46b5f8b0015 connected 2013-04-07 12:16:03,450 DEBUG [pool-96-thread-1] zookeeper.ZKUtil(1680): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e-0x13de46b5f8b0015 Retrieved 34 byte(s) of data from znode /hbase/running and set watcher; PBUF\x0A\x1CSun Apr 07 12:14:... 2013-04-07 12:16:03,451 DEBUG [RegionServer:3;asf001.sp2.ygridcore.net,49518,1365336962323-EventThread] zookeeper.ZooKeeperWatcher(274): regionserver:49518-0x13de46b5f8b0012 Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2013-04-07 12:16:03,451 DEBUG [RegionServer:0;asf001.sp2.ygridcore.net,60884,1365336878389-EventThread] zookeeper.ZooKeeperWatcher(274): regionserver:60884-0x13de46b5f8b0001 Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2013-04-07 12:16:03,454 DEBUG [RegionServer:3;asf001.sp2.ygridcore.net,49518,1365336962323-EventThread] zookeeper.ZKUtil(1680): regionserver:49518-0x13de46b5f8b0012 Retrieved 43 byte(s) of data from znode /hbase/master and set watcher; PBUF\x0A%\x0A\x18asf001.sp2.y... 2013-04-07 12:16:03,454 WARN [pool-96-thread-1] hbase.ZNodeClearer(58): No filename given to save the znode used, it won't be saved (Environment variable HBASE_ZNODE_FILE is not set). 2013-04-07 12:16:03,454 INFO [pool-96-thread-1] master.ActiveMasterManager(159): Master=asf001.sp2.ygridcore.net,33796,1365336878036 2013-04-07 12:16:03,455 INFO [pool-96-thread-1] master.SplitLogManager(181): timeout = 120000 2013-04-07 12:16:03,455 INFO [pool-96-thread-1] master.SplitLogManager(182): unassigned timeout = 180000 2013-04-07 12:16:03,455 DEBUG [RegionServer:0;asf001.sp2.ygridcore.net,60884,1365336878389-EventThread] zookeeper.ZKUtil(1680): regionserver:60884-0x13de46b5f8b0001 Retrieved 43 byte(s) of data from znode /hbase/master and set watcher; PBUF\x0A%\x0A\x18asf001.sp2.y... 2013-04-07 12:16:03,456 INFO [pool-96-thread-1] master.SplitLogManager(858): found 0 orphan tasks and 0 rescan nodes 2013-04-07 12:16:03,466 INFO [pool-96-thread-1] util.FSTableDescriptors(630): Current tableInfoPath = hdfs://localhost:60405/user/jenkins/hbase/.META./.tableinfo.0000000001 2013-04-07 12:16:03,469 INFO [pool-96-thread-1] util.FSTableDescriptors(634): TableInfo already exists.. Skipping creation 2013-04-07 12:16:03,475 DEBUG [pool-96-thread-1] fs.HFileSystem(199): Starting addLocationsOrderInterceptor with class class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2013-04-07 12:16:03,475 INFO [pool-96-thread-1] fs.HFileSystem(246): Added intercepting call to namenode#getBlockLocations 2013-04-07 12:16:03,478 DEBUG [pool-96-thread-1] catalog.CatalogTracker(192): Starting catalog tracker org.apache.hadoop.hbase.catalog.CatalogTracker@f92541 2013-04-07 12:16:03,478 DEBUG [pool-96-thread-1] zookeeper.ZKUtil(422): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e-0x13de46b5f8b0015 Set watcher on existing znode /hbase/meta-region-server 2013-04-07 12:16:03,479 DEBUG [pool-96-thread-1] zookeeper.ZKUtil(1680): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e-0x13de46b5f8b0015 Retrieved 43 byte(s) of data from znode /hbase/meta-region-server and set watcher; asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:16:03,480 DEBUG [pool-96-thread-1] zookeeper.ZKUtil(424): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e-0x13de46b5f8b0015 /hbase/balancer does not exist. Watcher is set. 2013-04-07 12:16:03,481 DEBUG [pool-96-thread-1] zookeeper.ZKUtil(1680): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e-0x13de46b5f8b0015 Retrieved 6 byte(s) of data from znode /hbase/table/.META.; data=PBUF\x08\x00 2013-04-07 12:16:03,482 DEBUG [pool-96-thread-1] zookeeper.ZKUtil(1680): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e-0x13de46b5f8b0015 Retrieved 6 byte(s) of data from znode /hbase/table/testRegionAssignmentAfterMasterRecoveryDueToZKExpiry; data=PBUF\x08\x00 2013-04-07 12:16:03,483 DEBUG [pool-96-thread-1] zookeeper.ZKUtil(1680): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e-0x13de46b5f8b0015 Retrieved 6 byte(s) of data from znode /hbase/table/test1365336888306; data=PBUF\x08\x00 2013-04-07 12:16:03,484 DEBUG [pool-96-thread-1] zookeeper.ZKUtil(1680): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e-0x13de46b5f8b0015 Retrieved 6 byte(s) of data from znode /hbase/table/testLogSplittingAfterMasterRecoveryDueToZKExpiry; data=PBUF\x08\x00 2013-04-07 12:16:03,484 DEBUG [pool-96-thread-1] zookeeper.ZKUtil(1680): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e-0x13de46b5f8b0015 Retrieved 6 byte(s) of data from znode /hbase/table/test1365336881340; data=PBUF\x08\x00 2013-04-07 12:16:03,486 DEBUG [pool-96-thread-1] zookeeper.ZKUtil(422): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e-0x13de46b5f8b0015 Set watcher on existing znode /hbase/rs/asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:16:03,486 DEBUG [pool-96-thread-1] zookeeper.ZKUtil(422): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e-0x13de46b5f8b0015 Set watcher on existing znode /hbase/rs/asf001.sp2.ygridcore.net,43802,1365336889418 2013-04-07 12:16:03,487 DEBUG [pool-96-thread-1] zookeeper.ZKUtil(422): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e-0x13de46b5f8b0015 Set watcher on existing znode /hbase/rs/asf001.sp2.ygridcore.net,49518,1365336962323 2013-04-07 12:16:03,488 INFO [pool-96-thread-1] master.HMaster(615): Server active/primary master; asf001.sp2.ygridcore.net,33796,1365336878036, sessionid=0x13de46b5f8b0015, cluster-up flag was=true 2013-04-07 12:16:03,490 INFO [pool-96-thread-1] zookeeper.RecoverableZooKeeper(460): Node /hbase/online-snapshot/acquired already exists and this is not a retry 2013-04-07 12:16:03,491 INFO [pool-96-thread-1] procedure.ZKProcedureUtil(266): Clearing all procedure znodes: /hbase/online-snapshot/acquired /hbase/online-snapshot/reached /hbase/online-snapshot/abort 2013-04-07 12:16:03,493 DEBUG [pool-96-thread-1] procedure.ZKProcedureCoordinatorRpcs(194): Starting the controller for procedure member:asf001.sp2.ygridcore.net,33796,1365336878036 2013-04-07 12:16:03,493 WARN [pool-96-thread-1] snapshot.SnapshotManager(242): Couldn't delete working snapshot directory: hdfs://localhost:60405/user/jenkins/hbase/.snapshot/.tmp 2013-04-07 12:16:03,493 INFO [pool-96-thread-1] master.ServerManager(795): Finished waiting for region servers count to settle; checked in 3, slept for 0 ms, expecting minimum of 1, maximum of 2, master is running. 2013-04-07 12:16:03,495 INFO [pool-96-thread-1] master.MasterFileSystem(252): Log folder hdfs://localhost:60405/user/jenkins/hbase/.logs/asf001.sp2.ygridcore.net,43802,1365336889418 belongs to an existing region server 2013-04-07 12:16:03,496 INFO [pool-96-thread-1] master.MasterFileSystem(252): Log folder hdfs://localhost:60405/user/jenkins/hbase/.logs/asf001.sp2.ygridcore.net,49518,1365336962323 belongs to an existing region server 2013-04-07 12:16:03,496 INFO [pool-96-thread-1] master.MasterFileSystem(252): Log folder hdfs://localhost:60405/user/jenkins/hbase/.logs/asf001.sp2.ygridcore.net,60884,1365336878389 belongs to an existing region server 2013-04-07 12:16:03,496 INFO [pool-96-thread-1] master.MasterFileSystem(346): No logs to split 2013-04-07 12:16:03,496 INFO [pool-96-thread-1] master.MasterFileSystem(346): No logs to split 2013-04-07 12:16:03,498 DEBUG [pool-96-thread-1] zookeeper.ZKUtil(1680): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e-0x13de46b5f8b0015 Retrieved 43 byte(s) of data from znode /hbase/meta-region-server and set watcher; asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:16:03,499 DEBUG [pool-96-thread-1] zookeeper.ZKUtil(1680): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e-0x13de46b5f8b0015 Retrieved 43 byte(s) of data from znode /hbase/meta-region-server and set watcher; asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:16:03,500 DEBUG [pool-96-thread-1] zookeeper.ZKUtil(1680): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e-0x13de46b5f8b0015 Retrieved 43 byte(s) of data from znode /hbase/meta-region-server and set watcher; asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:16:03,500 DEBUG [pool-96-thread-1] master.RegionStates(290): Online a region with current state=OFFLINE, expected state=OPEN, assigned to server: null expected asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:16:03,500 INFO [pool-96-thread-1] master.RegionStates(264): Region {NAME => '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192,} transitioned from {.META.,,1.1028785192 state=OFFLINE, ts=1365336963496, server=null} to {.META.,,1.1028785192 state=OPEN, ts=1365336963500, server=asf001.sp2.ygridcore.net,60884,1365336878389} 2013-04-07 12:16:03,501 INFO [pool-96-thread-1] master.RegionStates(299): Onlined region {NAME => '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192,} on asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:16:03,501 DEBUG [pool-96-thread-1] zookeeper.ZKUtil(1680): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e-0x13de46b5f8b0015 Retrieved 43 byte(s) of data from znode /hbase/meta-region-server and set watcher; asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:16:03,501 INFO [pool-96-thread-1] master.HMaster(897): .META. assigned=0, rit=false, location=asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:16:03,501 INFO [pool-96-thread-1] master.ServerManager(533): AssignmentManager hasn't finished failover cleanup 2013-04-07 12:16:03,502 DEBUG [pool-96-thread-1] zookeeper.ZKUtil(1680): hconnection-0x1f327e-0x13de46b5f8b0003 Retrieved 43 byte(s) of data from znode /hbase/meta-region-server; data=asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:16:03,502 DEBUG [pool-96-thread-1] client.ClientScanner(96): Creating scanner over .META. starting at key '' 2013-04-07 12:16:03,503 DEBUG [pool-96-thread-1] client.ClientScanner(209): Advancing internal scanner to startKey at '' 2013-04-07 12:16:03,503 DEBUG [pool-96-thread-1] zookeeper.ZKUtil(1680): hconnection-0x1f327e-0x13de46b5f8b0003 Retrieved 43 byte(s) of data from znode /hbase/meta-region-server; data=asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:16:03,512 DEBUG [pool-96-thread-1] client.ClientScanner(196): Finished with scanning at {NAME => '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192,} 2013-04-07 12:16:03,512 INFO [pool-96-thread-1] catalog.MetaMigrationConvertingToPB(132): META already up-to date with PB serialization 2013-04-07 12:16:03,514 DEBUG [pool-96-thread-1] zookeeper.ZKUtil(1680): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e-0x13de46b5f8b0015 Retrieved 6 byte(s) of data from znode /hbase/table/.META.; data=PBUF\x08\x00 2013-04-07 12:16:03,515 DEBUG [pool-96-thread-1] zookeeper.ZKUtil(1680): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e-0x13de46b5f8b0015 Retrieved 6 byte(s) of data from znode /hbase/table/testRegionAssignmentAfterMasterRecoveryDueToZKExpiry; data=PBUF\x08\x00 2013-04-07 12:16:03,516 DEBUG [pool-96-thread-1] zookeeper.ZKUtil(1680): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e-0x13de46b5f8b0015 Retrieved 6 byte(s) of data from znode /hbase/table/test1365336888306; data=PBUF\x08\x00 2013-04-07 12:16:03,516 DEBUG [pool-96-thread-1] zookeeper.ZKUtil(1680): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e-0x13de46b5f8b0015 Retrieved 6 byte(s) of data from znode /hbase/table/testLogSplittingAfterMasterRecoveryDueToZKExpiry; data=PBUF\x08\x00 2013-04-07 12:16:03,517 DEBUG [pool-96-thread-1] zookeeper.ZKUtil(1680): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e-0x13de46b5f8b0015 Retrieved 6 byte(s) of data from znode /hbase/table/test1365336881340; data=PBUF\x08\x00 2013-04-07 12:16:03,518 DEBUG [pool-96-thread-1] zookeeper.ZKUtil(1680): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e-0x13de46b5f8b0015 Retrieved 6 byte(s) of data from znode /hbase/table/.META.; data=PBUF\x08\x00 2013-04-07 12:16:03,519 DEBUG [pool-96-thread-1] zookeeper.ZKUtil(1680): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e-0x13de46b5f8b0015 Retrieved 6 byte(s) of data from znode /hbase/table/testRegionAssignmentAfterMasterRecoveryDueToZKExpiry; data=PBUF\x08\x00 2013-04-07 12:16:03,520 DEBUG [pool-96-thread-1] zookeeper.ZKUtil(1680): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e-0x13de46b5f8b0015 Retrieved 6 byte(s) of data from znode /hbase/table/test1365336888306; data=PBUF\x08\x00 2013-04-07 12:16:03,521 DEBUG [pool-96-thread-1] zookeeper.ZKUtil(1680): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e-0x13de46b5f8b0015 Retrieved 6 byte(s) of data from znode /hbase/table/testLogSplittingAfterMasterRecoveryDueToZKExpiry; data=PBUF\x08\x00 2013-04-07 12:16:03,521 DEBUG [pool-96-thread-1] zookeeper.ZKUtil(1680): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e-0x13de46b5f8b0015 Retrieved 6 byte(s) of data from znode /hbase/table/test1365336881340; data=PBUF\x08\x00 2013-04-07 12:16:03,523 DEBUG [pool-96-thread-1] zookeeper.ZKUtil(1680): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e-0x13de46b5f8b0015 Retrieved 6 byte(s) of data from znode /hbase/table/.META.; data=PBUF\x08\x00 2013-04-07 12:16:03,523 DEBUG [pool-96-thread-1] zookeeper.ZKUtil(1680): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e-0x13de46b5f8b0015 Retrieved 6 byte(s) of data from znode /hbase/table/testRegionAssignmentAfterMasterRecoveryDueToZKExpiry; data=PBUF\x08\x00 2013-04-07 12:16:03,524 DEBUG [pool-96-thread-1] zookeeper.ZKUtil(1680): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e-0x13de46b5f8b0015 Retrieved 6 byte(s) of data from znode /hbase/table/test1365336888306; data=PBUF\x08\x00 2013-04-07 12:16:03,526 DEBUG [pool-96-thread-1] zookeeper.ZKUtil(1680): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e-0x13de46b5f8b0015 Retrieved 6 byte(s) of data from znode /hbase/table/testLogSplittingAfterMasterRecoveryDueToZKExpiry; data=PBUF\x08\x00 2013-04-07 12:16:03,526 DEBUG [pool-96-thread-1] zookeeper.ZKUtil(1680): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e-0x13de46b5f8b0015 Retrieved 6 byte(s) of data from znode /hbase/table/test1365336881340; data=PBUF\x08\x00 2013-04-07 12:16:03,527 DEBUG [pool-96-thread-1] zookeeper.ZKUtil(1680): hconnection-0x1f327e-0x13de46b5f8b0003 Retrieved 43 byte(s) of data from znode /hbase/meta-region-server; data=asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:16:03,527 DEBUG [pool-96-thread-1] client.ClientScanner(96): Creating scanner over .META. starting at key '' 2013-04-07 12:16:03,527 DEBUG [pool-96-thread-1] client.ClientScanner(209): Advancing internal scanner to startKey at '' 2013-04-07 12:16:03,528 DEBUG [pool-96-thread-1] zookeeper.ZKUtil(1680): hconnection-0x1f327e-0x13de46b5f8b0003 Retrieved 43 byte(s) of data from znode /hbase/meta-region-server; data=asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:16:03,535 DEBUG [pool-96-thread-1] client.ClientScanner(196): Finished with scanning at {NAME => '.META.,,1', STARTKEY => '', ENDKEY => '', ENCODED => 1028785192,} 2013-04-07 12:16:03,536 DEBUG [pool-96-thread-1] master.RegionStates(290): Online a region with current state=OFFLINE, expected state=OPEN, assigned to server: null expected asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:16:03,536 INFO [pool-96-thread-1] master.RegionStates(264): Region {NAME => 'test1365336881340,,1365336881424.c9e9e308a02801bb891879b6cf471381.', STARTKEY => '', ENDKEY => '', ENCODED => c9e9e308a02801bb891879b6cf471381,} transitioned from {test1365336881340,,1365336881424.c9e9e308a02801bb891879b6cf471381. state=OFFLINE, ts=1365336963536, server=null} to {test1365336881340,,1365336881424.c9e9e308a02801bb891879b6cf471381. state=OPEN, ts=1365336963536, server=asf001.sp2.ygridcore.net,60884,1365336878389} 2013-04-07 12:16:03,536 INFO [pool-96-thread-1] master.RegionStates(299): Onlined region {NAME => 'test1365336881340,,1365336881424.c9e9e308a02801bb891879b6cf471381.', STARTKEY => '', ENDKEY => '', ENCODED => c9e9e308a02801bb891879b6cf471381,} on asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:16:03,536 DEBUG [pool-96-thread-1] master.RegionStates(290): Online a region with current state=OFFLINE, expected state=OPEN, assigned to server: null expected asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:16:03,536 INFO [pool-96-thread-1] master.RegionStates(264): Region {NAME => 'test1365336888306,,1365336888309.da419688e5c79938e207c0d3c056a9ed.', STARTKEY => '', ENDKEY => '', ENCODED => da419688e5c79938e207c0d3c056a9ed,} transitioned from {test1365336888306,,1365336888309.da419688e5c79938e207c0d3c056a9ed. state=OFFLINE, ts=1365336963536, server=null} to {test1365336888306,,1365336888309.da419688e5c79938e207c0d3c056a9ed. state=OPEN, ts=1365336963536, server=asf001.sp2.ygridcore.net,60884,1365336878389} 2013-04-07 12:16:03,536 INFO [pool-96-thread-1] master.RegionStates(299): Onlined region {NAME => 'test1365336888306,,1365336888309.da419688e5c79938e207c0d3c056a9ed.', STARTKEY => '', ENDKEY => '', ENCODED => da419688e5c79938e207c0d3c056a9ed,} on asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:16:03,537 DEBUG [pool-96-thread-1] master.RegionStates(290): Online a region with current state=OFFLINE, expected state=OPEN, assigned to server: null expected asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:16:03,537 INFO [pool-96-thread-1] master.RegionStates(264): Region {NAME => 'testLogSplittingAfterMasterRecoveryDueToZKExpiry,,1365336889784.fb4182aef4ce07f011871ae0a083aee0.', STARTKEY => '', ENDKEY => '1', ENCODED => fb4182aef4ce07f011871ae0a083aee0,} transitioned from {testLogSplittingAfterMasterRecoveryDueToZKExpiry,,1365336889784.fb4182aef4ce07f011871ae0a083aee0. state=OFFLINE, ts=1365336963537, server=null} to {testLogSplittingAfterMasterRecoveryDueToZKExpiry,,1365336889784.fb4182aef4ce07f011871ae0a083aee0. state=OPEN, ts=1365336963537, server=asf001.sp2.ygridcore.net,60884,1365336878389} 2013-04-07 12:16:03,537 INFO [pool-96-thread-1] master.RegionStates(299): Onlined region {NAME => 'testLogSplittingAfterMasterRecoveryDueToZKExpiry,,1365336889784.fb4182aef4ce07f011871ae0a083aee0.', STARTKEY => '', ENDKEY => '1', ENCODED => fb4182aef4ce07f011871ae0a083aee0,} on asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:16:03,537 DEBUG [pool-96-thread-1] master.RegionStates(290): Online a region with current state=OFFLINE, expected state=OPEN, assigned to server: null expected asf001.sp2.ygridcore.net,43802,1365336889418 2013-04-07 12:16:03,537 INFO [pool-96-thread-1] master.RegionStates(264): Region {NAME => 'testLogSplittingAfterMasterRecoveryDueToZKExpiry,1,1365336889784.43207168ac66dce85cf2a4f916007450.', STARTKEY => '1', ENDKEY => '2', ENCODED => 43207168ac66dce85cf2a4f916007450,} transitioned from {testLogSplittingAfterMasterRecoveryDueToZKExpiry,1,1365336889784.43207168ac66dce85cf2a4f916007450. state=OFFLINE, ts=1365336963537, server=null} to {testLogSplittingAfterMasterRecoveryDueToZKExpiry,1,1365336889784.43207168ac66dce85cf2a4f916007450. state=OPEN, ts=1365336963537, server=asf001.sp2.ygridcore.net,43802,1365336889418} 2013-04-07 12:16:03,537 INFO [pool-96-thread-1] master.RegionStates(299): Onlined region {NAME => 'testLogSplittingAfterMasterRecoveryDueToZKExpiry,1,1365336889784.43207168ac66dce85cf2a4f916007450.', STARTKEY => '1', ENDKEY => '2', ENCODED => 43207168ac66dce85cf2a4f916007450,} on asf001.sp2.ygridcore.net,43802,1365336889418 2013-04-07 12:16:03,538 DEBUG [pool-96-thread-1] master.RegionStates(290): Online a region with current state=OFFLINE, expected state=OPEN, assigned to server: null expected asf001.sp2.ygridcore.net,43802,1365336889418 2013-04-07 12:16:03,538 INFO [pool-96-thread-1] master.RegionStates(264): Region {NAME => 'testLogSplittingAfterMasterRecoveryDueToZKExpiry,2,1365336889784.55884b67b9203a71bdf8c0e3de70452d.', STARTKEY => '2', ENDKEY => '3', ENCODED => 55884b67b9203a71bdf8c0e3de70452d,} transitioned from {testLogSplittingAfterMasterRecoveryDueToZKExpiry,2,1365336889784.55884b67b9203a71bdf8c0e3de70452d. state=OFFLINE, ts=1365336963538, server=null} to {testLogSplittingAfterMasterRecoveryDueToZKExpiry,2,1365336889784.55884b67b9203a71bdf8c0e3de70452d. state=OPEN, ts=1365336963538, server=asf001.sp2.ygridcore.net,43802,1365336889418} 2013-04-07 12:16:03,538 INFO [pool-96-thread-1] master.RegionStates(299): Onlined region {NAME => 'testLogSplittingAfterMasterRecoveryDueToZKExpiry,2,1365336889784.55884b67b9203a71bdf8c0e3de70452d.', STARTKEY => '2', ENDKEY => '3', ENCODED => 55884b67b9203a71bdf8c0e3de70452d,} on asf001.sp2.ygridcore.net,43802,1365336889418 2013-04-07 12:16:03,538 DEBUG [pool-96-thread-1] master.RegionStates(290): Online a region with current state=OFFLINE, expected state=OPEN, assigned to server: null expected asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:16:03,538 INFO [pool-96-thread-1] master.RegionStates(264): Region {NAME => 'testLogSplittingAfterMasterRecoveryDueToZKExpiry,3,1365336889784.4e9a3d63237d29e6bf55bf2360834cbe.', STARTKEY => '3', ENDKEY => '4', ENCODED => 4e9a3d63237d29e6bf55bf2360834cbe,} transitioned from {testLogSplittingAfterMasterRecoveryDueToZKExpiry,3,1365336889784.4e9a3d63237d29e6bf55bf2360834cbe. state=OFFLINE, ts=1365336963538, server=null} to {testLogSplittingAfterMasterRecoveryDueToZKExpiry,3,1365336889784.4e9a3d63237d29e6bf55bf2360834cbe. state=OPEN, ts=1365336963538, server=asf001.sp2.ygridcore.net,60884,1365336878389} 2013-04-07 12:16:03,538 INFO [pool-96-thread-1] master.RegionStates(299): Onlined region {NAME => 'testLogSplittingAfterMasterRecoveryDueToZKExpiry,3,1365336889784.4e9a3d63237d29e6bf55bf2360834cbe.', STARTKEY => '3', ENDKEY => '4', ENCODED => 4e9a3d63237d29e6bf55bf2360834cbe,} on asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:16:03,539 DEBUG [pool-96-thread-1] master.RegionStates(290): Online a region with current state=OFFLINE, expected state=OPEN, assigned to server: null expected asf001.sp2.ygridcore.net,43802,1365336889418 2013-04-07 12:16:03,539 INFO [pool-96-thread-1] master.RegionStates(264): Region {NAME => 'testLogSplittingAfterMasterRecoveryDueToZKExpiry,4,1365336889784.567d8ef20f90f3c1b8a889d22a1c3948.', STARTKEY => '4', ENDKEY => '5', ENCODED => 567d8ef20f90f3c1b8a889d22a1c3948,} transitioned from {testLogSplittingAfterMasterRecoveryDueToZKExpiry,4,1365336889784.567d8ef20f90f3c1b8a889d22a1c3948. state=OFFLINE, ts=1365336963539, server=null} to {testLogSplittingAfterMasterRecoveryDueToZKExpiry,4,1365336889784.567d8ef20f90f3c1b8a889d22a1c3948. state=OPEN, ts=1365336963539, server=asf001.sp2.ygridcore.net,43802,1365336889418} 2013-04-07 12:16:03,539 INFO [pool-96-thread-1] master.RegionStates(299): Onlined region {NAME => 'testLogSplittingAfterMasterRecoveryDueToZKExpiry,4,1365336889784.567d8ef20f90f3c1b8a889d22a1c3948.', STARTKEY => '4', ENDKEY => '5', ENCODED => 567d8ef20f90f3c1b8a889d22a1c3948,} on asf001.sp2.ygridcore.net,43802,1365336889418 2013-04-07 12:16:03,539 DEBUG [pool-96-thread-1] master.RegionStates(290): Online a region with current state=OFFLINE, expected state=OPEN, assigned to server: null expected asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:16:03,539 INFO [pool-96-thread-1] master.RegionStates(264): Region {NAME => 'testLogSplittingAfterMasterRecoveryDueToZKExpiry,5,1365336889784.5403a557a6168e5f3c5141eb494f6088.', STARTKEY => '5', ENDKEY => '', ENCODED => 5403a557a6168e5f3c5141eb494f6088,} transitioned from {testLogSplittingAfterMasterRecoveryDueToZKExpiry,5,1365336889784.5403a557a6168e5f3c5141eb494f6088. state=OFFLINE, ts=1365336963539, server=null} to {testLogSplittingAfterMasterRecoveryDueToZKExpiry,5,1365336889784.5403a557a6168e5f3c5141eb494f6088. state=OPEN, ts=1365336963539, server=asf001.sp2.ygridcore.net,60884,1365336878389} 2013-04-07 12:16:03,539 INFO [pool-96-thread-1] master.RegionStates(299): Onlined region {NAME => 'testLogSplittingAfterMasterRecoveryDueToZKExpiry,5,1365336889784.5403a557a6168e5f3c5141eb494f6088.', STARTKEY => '5', ENDKEY => '', ENCODED => 5403a557a6168e5f3c5141eb494f6088,} on asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:16:03,540 DEBUG [pool-96-thread-1] master.RegionStates(290): Online a region with current state=OFFLINE, expected state=OPEN, assigned to server: null expected asf001.sp2.ygridcore.net,43802,1365336889418 2013-04-07 12:16:03,540 INFO [pool-96-thread-1] master.RegionStates(264): Region {NAME => 'testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,,1365336962397.d4e129e51e67d798e17083623fdde133.', STARTKEY => '', ENDKEY => 'a', ENCODED => d4e129e51e67d798e17083623fdde133,} transitioned from {testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,,1365336962397.d4e129e51e67d798e17083623fdde133. state=OFFLINE, ts=1365336963540, server=null} to {testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,,1365336962397.d4e129e51e67d798e17083623fdde133. state=OPEN, ts=1365336963540, server=asf001.sp2.ygridcore.net,43802,1365336889418} 2013-04-07 12:16:03,540 INFO [pool-96-thread-1] master.RegionStates(299): Onlined region {NAME => 'testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,,1365336962397.d4e129e51e67d798e17083623fdde133.', STARTKEY => '', ENDKEY => 'a', ENCODED => d4e129e51e67d798e17083623fdde133,} on asf001.sp2.ygridcore.net,43802,1365336889418 2013-04-07 12:16:03,540 DEBUG [pool-96-thread-1] master.RegionStates(290): Online a region with current state=OFFLINE, expected state=OPEN, assigned to server: null expected asf001.sp2.ygridcore.net,49518,1365336962323 2013-04-07 12:16:03,540 INFO [pool-96-thread-1] master.RegionStates(264): Region {NAME => 'testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,a,1365336962398.b1b340dff1e06fd6cb0b0a9c64964551.', STARTKEY => 'a', ENDKEY => 'b', ENCODED => b1b340dff1e06fd6cb0b0a9c64964551,} transitioned from {testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,a,1365336962398.b1b340dff1e06fd6cb0b0a9c64964551. state=OFFLINE, ts=1365336963540, server=null} to {testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,a,1365336962398.b1b340dff1e06fd6cb0b0a9c64964551. state=OPEN, ts=1365336963540, server=asf001.sp2.ygridcore.net,49518,1365336962323} 2013-04-07 12:16:03,540 INFO [pool-96-thread-1] master.RegionStates(299): Onlined region {NAME => 'testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,a,1365336962398.b1b340dff1e06fd6cb0b0a9c64964551.', STARTKEY => 'a', ENDKEY => 'b', ENCODED => b1b340dff1e06fd6cb0b0a9c64964551,} on asf001.sp2.ygridcore.net,49518,1365336962323 2013-04-07 12:16:03,541 DEBUG [pool-96-thread-1] master.RegionStates(290): Online a region with current state=OFFLINE, expected state=OPEN, assigned to server: null expected asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:16:03,541 INFO [pool-96-thread-1] master.RegionStates(264): Region {NAME => 'testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,b,1365336962398.11c8123e8d563f264d947ffa106e4c7c.', STARTKEY => 'b', ENDKEY => 'c', ENCODED => 11c8123e8d563f264d947ffa106e4c7c,} transitioned from {testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,b,1365336962398.11c8123e8d563f264d947ffa106e4c7c. state=OFFLINE, ts=1365336963541, server=null} to {testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,b,1365336962398.11c8123e8d563f264d947ffa106e4c7c. state=OPEN, ts=1365336963541, server=asf001.sp2.ygridcore.net,60884,1365336878389} 2013-04-07 12:16:03,541 INFO [pool-96-thread-1] master.RegionStates(299): Onlined region {NAME => 'testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,b,1365336962398.11c8123e8d563f264d947ffa106e4c7c.', STARTKEY => 'b', ENDKEY => 'c', ENCODED => 11c8123e8d563f264d947ffa106e4c7c,} on asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:16:03,541 DEBUG [pool-96-thread-1] master.RegionStates(290): Online a region with current state=OFFLINE, expected state=OPEN, assigned to server: null expected asf001.sp2.ygridcore.net,43802,1365336889418 2013-04-07 12:16:03,541 INFO [pool-96-thread-1] master.RegionStates(264): Region {NAME => 'testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,c,1365336962398.4432e800611c00a4ae168862a273fc70.', STARTKEY => 'c', ENDKEY => 'd', ENCODED => 4432e800611c00a4ae168862a273fc70,} transitioned from {testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,c,1365336962398.4432e800611c00a4ae168862a273fc70. state=OFFLINE, ts=1365336963541, server=null} to {testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,c,1365336962398.4432e800611c00a4ae168862a273fc70. state=OPEN, ts=1365336963541, server=asf001.sp2.ygridcore.net,43802,1365336889418} 2013-04-07 12:16:03,541 INFO [pool-96-thread-1] master.RegionStates(299): Onlined region {NAME => 'testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,c,1365336962398.4432e800611c00a4ae168862a273fc70.', STARTKEY => 'c', ENDKEY => 'd', ENCODED => 4432e800611c00a4ae168862a273fc70,} on asf001.sp2.ygridcore.net,43802,1365336889418 2013-04-07 12:16:03,541 DEBUG [pool-96-thread-1] master.RegionStates(290): Online a region with current state=OFFLINE, expected state=OPEN, assigned to server: null expected asf001.sp2.ygridcore.net,49518,1365336962323 2013-04-07 12:16:03,542 INFO [pool-96-thread-1] master.RegionStates(264): Region {NAME => 'testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,d,1365336962398.e99798d2c4d566ddb063efe03842457f.', STARTKEY => 'd', ENDKEY => 'e', ENCODED => e99798d2c4d566ddb063efe03842457f,} transitioned from {testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,d,1365336962398.e99798d2c4d566ddb063efe03842457f. state=OFFLINE, ts=1365336963541, server=null} to {testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,d,1365336962398.e99798d2c4d566ddb063efe03842457f. state=OPEN, ts=1365336963542, server=asf001.sp2.ygridcore.net,49518,1365336962323} 2013-04-07 12:16:03,542 INFO [pool-96-thread-1] master.RegionStates(299): Onlined region {NAME => 'testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,d,1365336962398.e99798d2c4d566ddb063efe03842457f.', STARTKEY => 'd', ENDKEY => 'e', ENCODED => e99798d2c4d566ddb063efe03842457f,} on asf001.sp2.ygridcore.net,49518,1365336962323 2013-04-07 12:16:03,542 DEBUG [pool-96-thread-1] master.RegionStates(290): Online a region with current state=OFFLINE, expected state=OPEN, assigned to server: null expected asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:16:03,542 INFO [pool-96-thread-1] master.RegionStates(264): Region {NAME => 'testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,e,1365336962398.dbb82f7f432d6664327da0b8f15b7d76.', STARTKEY => 'e', ENDKEY => 'f', ENCODED => dbb82f7f432d6664327da0b8f15b7d76,} transitioned from {testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,e,1365336962398.dbb82f7f432d6664327da0b8f15b7d76. state=OFFLINE, ts=1365336963542, server=null} to {testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,e,1365336962398.dbb82f7f432d6664327da0b8f15b7d76. state=OPEN, ts=1365336963542, server=asf001.sp2.ygridcore.net,60884,1365336878389} 2013-04-07 12:16:03,542 INFO [pool-96-thread-1] master.RegionStates(299): Onlined region {NAME => 'testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,e,1365336962398.dbb82f7f432d6664327da0b8f15b7d76.', STARTKEY => 'e', ENDKEY => 'f', ENCODED => dbb82f7f432d6664327da0b8f15b7d76,} on asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:16:03,542 DEBUG [pool-96-thread-1] master.RegionStates(290): Online a region with current state=OFFLINE, expected state=OPEN, assigned to server: null expected asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:16:03,543 INFO [pool-96-thread-1] master.RegionStates(264): Region {NAME => 'testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,f,1365336962398.eff8376859f5e6ba3201ec0ec9536d0f.', STARTKEY => 'f', ENDKEY => 'g', ENCODED => eff8376859f5e6ba3201ec0ec9536d0f,} transitioned from {testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,f,1365336962398.eff8376859f5e6ba3201ec0ec9536d0f. state=OFFLINE, ts=1365336963542, server=null} to {testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,f,1365336962398.eff8376859f5e6ba3201ec0ec9536d0f. state=OPEN, ts=1365336963543, server=asf001.sp2.ygridcore.net,60884,1365336878389} 2013-04-07 12:16:03,543 INFO [pool-96-thread-1] master.RegionStates(299): Onlined region {NAME => 'testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,f,1365336962398.eff8376859f5e6ba3201ec0ec9536d0f.', STARTKEY => 'f', ENDKEY => 'g', ENCODED => eff8376859f5e6ba3201ec0ec9536d0f,} on asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:16:03,543 DEBUG [pool-96-thread-1] master.RegionStates(290): Online a region with current state=OFFLINE, expected state=OPEN, assigned to server: null expected asf001.sp2.ygridcore.net,49518,1365336962323 2013-04-07 12:16:03,543 INFO [pool-96-thread-1] master.RegionStates(264): Region {NAME => 'testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,g,1365336962398.682475e1358b7306bd3406e9330263a4.', STARTKEY => 'g', ENDKEY => 'h', ENCODED => 682475e1358b7306bd3406e9330263a4,} transitioned from {testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,g,1365336962398.682475e1358b7306bd3406e9330263a4. state=OFFLINE, ts=1365336963543, server=null} to {testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,g,1365336962398.682475e1358b7306bd3406e9330263a4. state=OPEN, ts=1365336963543, server=asf001.sp2.ygridcore.net,49518,1365336962323} 2013-04-07 12:16:03,543 INFO [pool-96-thread-1] master.RegionStates(299): Onlined region {NAME => 'testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,g,1365336962398.682475e1358b7306bd3406e9330263a4.', STARTKEY => 'g', ENDKEY => 'h', ENCODED => 682475e1358b7306bd3406e9330263a4,} on asf001.sp2.ygridcore.net,49518,1365336962323 2013-04-07 12:16:03,543 DEBUG [pool-96-thread-1] master.RegionStates(290): Online a region with current state=OFFLINE, expected state=OPEN, assigned to server: null expected asf001.sp2.ygridcore.net,43802,1365336889418 2013-04-07 12:16:03,544 INFO [pool-96-thread-1] master.RegionStates(264): Region {NAME => 'testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,h,1365336962398.1ce6d62427641e2a0e135d3d27b54b5d.', STARTKEY => 'h', ENDKEY => 'i', ENCODED => 1ce6d62427641e2a0e135d3d27b54b5d,} transitioned from {testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,h,1365336962398.1ce6d62427641e2a0e135d3d27b54b5d. state=OFFLINE, ts=1365336963543, server=null} to {testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,h,1365336962398.1ce6d62427641e2a0e135d3d27b54b5d. state=OPEN, ts=1365336963544, server=asf001.sp2.ygridcore.net,43802,1365336889418} 2013-04-07 12:16:03,544 INFO [pool-96-thread-1] master.RegionStates(299): Onlined region {NAME => 'testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,h,1365336962398.1ce6d62427641e2a0e135d3d27b54b5d.', STARTKEY => 'h', ENDKEY => 'i', ENCODED => 1ce6d62427641e2a0e135d3d27b54b5d,} on asf001.sp2.ygridcore.net,43802,1365336889418 2013-04-07 12:16:03,544 DEBUG [pool-96-thread-1] master.RegionStates(290): Online a region with current state=OFFLINE, expected state=OPEN, assigned to server: null expected asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:16:03,544 INFO [pool-96-thread-1] master.RegionStates(264): Region {NAME => 'testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,i,1365336962398.385e5846630b0cb4493758961a2c2457.', STARTKEY => 'i', ENDKEY => 'j', ENCODED => 385e5846630b0cb4493758961a2c2457,} transitioned from {testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,i,1365336962398.385e5846630b0cb4493758961a2c2457. state=OFFLINE, ts=1365336963544, server=null} to {testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,i,1365336962398.385e5846630b0cb4493758961a2c2457. state=OPEN, ts=1365336963544, server=asf001.sp2.ygridcore.net,60884,1365336878389} 2013-04-07 12:16:03,544 INFO [pool-96-thread-1] master.RegionStates(299): Onlined region {NAME => 'testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,i,1365336962398.385e5846630b0cb4493758961a2c2457.', STARTKEY => 'i', ENDKEY => 'j', ENCODED => 385e5846630b0cb4493758961a2c2457,} on asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:16:03,544 DEBUG [pool-96-thread-1] master.RegionStates(290): Online a region with current state=OFFLINE, expected state=OPEN, assigned to server: null expected asf001.sp2.ygridcore.net,49518,1365336962323 2013-04-07 12:16:03,545 INFO [pool-96-thread-1] master.RegionStates(264): Region {NAME => 'testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,j,1365336962398.38f0eb146fb473b19541deaada3419b9.', STARTKEY => 'j', ENDKEY => '', ENCODED => 38f0eb146fb473b19541deaada3419b9,} transitioned from {testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,j,1365336962398.38f0eb146fb473b19541deaada3419b9. state=OFFLINE, ts=1365336963544, server=null} to {testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,j,1365336962398.38f0eb146fb473b19541deaada3419b9. state=OPEN, ts=1365336963545, server=asf001.sp2.ygridcore.net,49518,1365336962323} 2013-04-07 12:16:03,545 INFO [pool-96-thread-1] master.RegionStates(299): Onlined region {NAME => 'testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,j,1365336962398.38f0eb146fb473b19541deaada3419b9.', STARTKEY => 'j', ENDKEY => '', ENCODED => 38f0eb146fb473b19541deaada3419b9,} on asf001.sp2.ygridcore.net,49518,1365336962323 2013-04-07 12:16:03,546 INFO [pool-96-thread-1] master.AssignmentManager(439): Found regions out on cluster or in RIT; failover 2013-04-07 12:16:03,547 DEBUG [pool-96-thread-1] zookeeper.ZKUtil(1680): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e-0x13de46b5f8b0015 Retrieved 6 byte(s) of data from znode /hbase/table/.META.; data=PBUF\x08\x00 2013-04-07 12:16:03,548 DEBUG [pool-96-thread-1] zookeeper.ZKUtil(1680): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e-0x13de46b5f8b0015 Retrieved 6 byte(s) of data from znode /hbase/table/testRegionAssignmentAfterMasterRecoveryDueToZKExpiry; data=PBUF\x08\x00 2013-04-07 12:16:03,549 DEBUG [pool-96-thread-1] zookeeper.ZKUtil(1680): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e-0x13de46b5f8b0015 Retrieved 6 byte(s) of data from znode /hbase/table/test1365336888306; data=PBUF\x08\x00 2013-04-07 12:16:03,550 DEBUG [pool-96-thread-1] zookeeper.ZKUtil(1680): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e-0x13de46b5f8b0015 Retrieved 6 byte(s) of data from znode /hbase/table/testLogSplittingAfterMasterRecoveryDueToZKExpiry; data=PBUF\x08\x00 2013-04-07 12:16:03,550 DEBUG [pool-96-thread-1] zookeeper.ZKUtil(1680): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e-0x13de46b5f8b0015 Retrieved 6 byte(s) of data from znode /hbase/table/test1365336881340; data=PBUF\x08\x00 2013-04-07 12:16:03,552 DEBUG [pool-96-thread-1] zookeeper.ZKUtil(1680): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e-0x13de46b5f8b0015 Retrieved 6 byte(s) of data from znode /hbase/table/.META.; data=PBUF\x08\x00 2013-04-07 12:16:03,553 DEBUG [pool-96-thread-1] zookeeper.ZKUtil(1680): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e-0x13de46b5f8b0015 Retrieved 6 byte(s) of data from znode /hbase/table/testRegionAssignmentAfterMasterRecoveryDueToZKExpiry; data=PBUF\x08\x00 2013-04-07 12:16:03,553 DEBUG [pool-96-thread-1] zookeeper.ZKUtil(1680): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e-0x13de46b5f8b0015 Retrieved 6 byte(s) of data from znode /hbase/table/test1365336888306; data=PBUF\x08\x00 2013-04-07 12:16:03,554 DEBUG [pool-96-thread-1] zookeeper.ZKUtil(1680): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e-0x13de46b5f8b0015 Retrieved 6 byte(s) of data from znode /hbase/table/testLogSplittingAfterMasterRecoveryDueToZKExpiry; data=PBUF\x08\x00 2013-04-07 12:16:03,555 DEBUG [pool-96-thread-1] zookeeper.ZKUtil(1680): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e-0x13de46b5f8b0015 Retrieved 6 byte(s) of data from znode /hbase/table/test1365336881340; data=PBUF\x08\x00 2013-04-07 12:16:03,555 INFO [pool-96-thread-1] master.HMaster(792): Master has completed initialization 2013-04-07 12:16:03,581 INFO [pool-1-thread-1] hbase.ResourceChecker(171): after: TestZooKeeper#testRegionAssignmentAfterMasterRecoveryDueToZKExpiry Thread=333 (was 226) Potentially hanging thread: RS_CLOSE_REGION-asf001.sp2.ygridcore.net,49518,1365336962323-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907) java.lang.Thread.run(Thread.java:662) Potentially hanging thread: hbase-am-zkevent-worker-pool-20-thread-35 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:424) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907) java.lang.Thread.run(Thread.java:662) Potentially hanging thread: RS_CLOSE_REGION-asf001.sp2.ygridcore.net,49518,1365336962323-1 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907) java.lang.Thread.run(Thread.java:662) Potentially hanging thread: hbase-am-zkevent-worker-pool-20-thread-23 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:424) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907) java.lang.Thread.run(Thread.java:662) Potentially hanging thread: RS_OPEN_REGION-asf001.sp2.ygridcore.net,49518,1365336962323-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907) java.lang.Thread.run(Thread.java:662) Potentially hanging thread: hbase-am-zkevent-worker-pool-2-thread-23 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:424) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907) java.lang.Thread.run(Thread.java:662) Potentially hanging thread: hbase-am-zkevent-worker-pool-20-thread-30 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:424) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907) java.lang.Thread.run(Thread.java:662) Potentially hanging thread: IPC Client (47) connection to asf001.sp2.ygridcore.net/67.195.138.31:60884 from jenkins.hfs.3 java.lang.Object.wait(Native Method) org.apache.hadoop.hbase.ipc.HBaseClient$Connection.waitForWork(HBaseClient.java:614) org.apache.hadoop.hbase.ipc.HBaseClient$Connection.run(HBaseClient.java:660) Potentially hanging thread: hbase-am-zkevent-worker-pool-2-thread-35 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:424) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907) java.lang.Thread.run(Thread.java:662) Potentially hanging thread: hbase-am-zkevent-worker-pool-20-thread-39 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:424) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907) java.lang.Thread.run(Thread.java:662) Potentially hanging thread: IPC Reader 0 on port 49518 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84) org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.doRunLoop(HBaseServer.java:582) org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:568) java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) java.lang.Thread.run(Thread.java:662) Potentially hanging thread: hbase-am-zkevent-worker-pool-2-thread-25 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:424) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907) java.lang.Thread.run(Thread.java:662) Potentially hanging thread: IPC Reader 7 on port 49518 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84) org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.doRunLoop(HBaseServer.java:582) org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:568) java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) java.lang.Thread.run(Thread.java:662) Potentially hanging thread: REPL IPC Server handler 0 on 49518 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399) org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1846) Potentially hanging thread: hbase-am-zkevent-worker-pool-20-thread-22 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:424) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907) java.lang.Thread.run(Thread.java:662) Potentially hanging thread: PRI IPC Server handler 9 on 49518 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399) org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1846) Potentially hanging thread: PRI IPC Server handler 7 on 49518 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399) org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1846) Potentially hanging thread: hbase-am-zkevent-worker-pool-2-thread-34 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:424) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907) java.lang.Thread.run(Thread.java:662) Potentially hanging thread: hbase-am-zkevent-worker-pool-20-thread-25 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:424) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907) java.lang.Thread.run(Thread.java:662) Potentially hanging thread: org.apache.hadoop.hdfs.LeaseRenewer$1@114a41 java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.LeaseRenewer.run(LeaseRenewer.java:397) org.apache.hadoop.hdfs.LeaseRenewer.access$600(LeaseRenewer.java:69) org.apache.hadoop.hdfs.LeaseRenewer$1.run(LeaseRenewer.java:273) java.lang.Thread.run(Thread.java:662) Potentially hanging thread: IPC Server handler 2 on 49518 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399) org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1846) Potentially hanging thread: hbase-am-zkevent-worker-pool-20-thread-37 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:424) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907) java.lang.Thread.run(Thread.java:662) Potentially hanging thread: hbase-am-zkevent-worker-pool-20-thread-24 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:424) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907) java.lang.Thread.run(Thread.java:662) Potentially hanging thread: Thread-301 java.lang.Object.wait(Native Method) org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2966) Potentially hanging thread: PRI IPC Server handler 5 on 49518 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399) org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1846) Potentially hanging thread: hbase-am-zkevent-worker-pool-2-thread-40 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:424) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907) java.lang.Thread.run(Thread.java:662) Potentially hanging thread: RS_OPEN_REGION-asf001.sp2.ygridcore.net,49518,1365336962323-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907) java.lang.Thread.run(Thread.java:662) Potentially hanging thread: hbase-am-zkevent-worker-pool-20-thread-27 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:424) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907) java.lang.Thread.run(Thread.java:662) Potentially hanging thread: PRI IPC Server handler 6 on 49518 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399) org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1846) Potentially hanging thread: hbase-am-zkevent-worker-pool-2-thread-38 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:424) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907) java.lang.Thread.run(Thread.java:662) Potentially hanging thread: hbase-am-zkevent-worker-pool-20-thread-40 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:424) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907) java.lang.Thread.run(Thread.java:662) Potentially hanging thread: hbase-am-zkevent-worker-pool-2-thread-31 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:424) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907) java.lang.Thread.run(Thread.java:662) Potentially hanging thread: RS_OPEN_REGION-asf001.sp2.ygridcore.net,49518,1365336962323-1 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907) java.lang.Thread.run(Thread.java:662) Potentially hanging thread: RegionServer:3;asf001.sp2.ygridcore.net,49518,1365336962323-EventThread sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399) org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:491) Potentially hanging thread: IPC Reader 1 on port 49518 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84) org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.doRunLoop(HBaseServer.java:582) org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:568) java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) java.lang.Thread.run(Thread.java:662) Potentially hanging thread: hbase-am-zkevent-worker-pool-2-thread-32 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:424) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907) java.lang.Thread.run(Thread.java:662) Potentially hanging thread: hbase-am-zkevent-worker-pool-20-thread-21 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:424) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907) java.lang.Thread.run(Thread.java:662) Potentially hanging thread: RegionServer:3;asf001.sp2.ygridcore.net,49518,1365336962323.compactionChecker java.lang.Object.wait(Native Method) org.apache.hadoop.hbase.util.Sleeper.sleep(Sleeper.java:94) org.apache.hadoop.hbase.Chore.run(Chore.java:89) java.lang.Thread.run(Thread.java:662) Potentially hanging thread: IPC Server handler 3 on 49518 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399) org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1846) Potentially hanging thread: REPL IPC Server handler 2 on 49518 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399) org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1846) Potentially hanging thread: hbase-am-zkevent-worker-pool-2-thread-36 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:424) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907) java.lang.Thread.run(Thread.java:662) Potentially hanging thread: hbase-am-zkevent-worker-pool-2-thread-39 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:424) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907) java.lang.Thread.run(Thread.java:662) Potentially hanging thread: hbase-am-zkevent-worker-pool-20-thread-34 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:424) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907) java.lang.Thread.run(Thread.java:662) Potentially hanging thread: SplitLogWorker-asf001.sp2.ygridcore.net,49518,1365336962323 java.lang.Object.wait(Native Method) java.lang.Object.wait(Object.java:485) org.apache.hadoop.hbase.regionserver.SplitLogWorker.taskLoop(SplitLogWorker.java:203) org.apache.hadoop.hbase.regionserver.SplitLogWorker.run(SplitLogWorker.java:162) java.lang.Thread.run(Thread.java:662) Potentially hanging thread: hbase-am-zkevent-worker-pool-2-thread-28 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:424) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907) java.lang.Thread.run(Thread.java:662) Potentially hanging thread: IPC Server listener on 49518 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84) org.apache.hadoop.hbase.ipc.HBaseServer$Listener.run(HBaseServer.java:690) Potentially hanging thread: org.apache.hadoop.hdfs.LeaseRenewer$1@dc135d java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.LeaseRenewer.run(LeaseRenewer.java:397) org.apache.hadoop.hdfs.LeaseRenewer.access$600(LeaseRenewer.java:69) org.apache.hadoop.hdfs.LeaseRenewer$1.run(LeaseRenewer.java:273) java.lang.Thread.run(Thread.java:662) Potentially hanging thread: hbase-am-zkevent-worker-pool-2-thread-21 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:424) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907) java.lang.Thread.run(Thread.java:662) Potentially hanging thread: hbase-am-zkevent-worker-pool-2-thread-22 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:424) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907) java.lang.Thread.run(Thread.java:662) Potentially hanging thread: RegionServer:3;asf001.sp2.ygridcore.net,49518,1365336962323.leaseChecker sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025) java.util.concurrent.DelayQueue.poll(DelayQueue.java:201) org.apache.hadoop.hbase.regionserver.Leases.run(Leases.java:82) java.lang.Thread.run(Thread.java:662) Potentially hanging thread: hbase-am-zkevent-worker-pool-20-thread-38 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:424) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907) java.lang.Thread.run(Thread.java:662) Potentially hanging thread: IPC Reader 6 on port 49518 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84) org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.doRunLoop(HBaseServer.java:582) org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:568) java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) java.lang.Thread.run(Thread.java:662) Potentially hanging thread: RS_CLOSE_REGION-asf001.sp2.ygridcore.net,60884,1365336878389-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907) java.lang.Thread.run(Thread.java:662) Potentially hanging thread: RegionServer:3;asf001.sp2.ygridcore.net,49518,1365336962323 java.lang.Object.wait(Native Method) org.apache.hadoop.hbase.util.Sleeper.sleep(Sleeper.java:94) org.apache.hadoop.hbase.util.Sleeper.sleep(Sleeper.java:58) org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:772) org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.runRegionServer(MiniHBaseCluster.java:151) org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.access$000(MiniHBaseCluster.java:103) org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer$1.run(MiniHBaseCluster.java:135) java.security.AccessController.doPrivileged(Native Method) javax.security.auth.Subject.doAs(Subject.java:337) org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1131) sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) java.lang.reflect.Method.invoke(Method.java:597) org.apache.hadoop.hbase.util.Methods.call(Methods.java:41) org.apache.hadoop.hbase.security.User.call(User.java:420) org.apache.hadoop.hbase.security.User.access$300(User.java:51) org.apache.hadoop.hbase.security.User$SecureHadoopUser.runAs(User.java:260) org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.run(MiniHBaseCluster.java:133) java.lang.Thread.run(Thread.java:662) Potentially hanging thread: IPC Reader 2 on port 49518 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84) org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.doRunLoop(HBaseServer.java:582) org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:568) java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) java.lang.Thread.run(Thread.java:662) Potentially hanging thread: hbase-am-zkevent-worker-pool-2-thread-33 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:424) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907) java.lang.Thread.run(Thread.java:662) Potentially hanging thread: Thread-303 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025) java.util.concurrent.DelayQueue.poll(DelayQueue.java:201) java.util.concurrent.DelayQueue.poll(DelayQueue.java:39) org.apache.hadoop.hbase.regionserver.MemStoreFlusher$FlushHandler.run(MemStoreFlusher.java:226) java.lang.Thread.run(Thread.java:662) Potentially hanging thread: PRI IPC Server handler 1 on 49518 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399) org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1846) Potentially hanging thread: PRI IPC Server handler 0 on 49518 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399) org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1846) Potentially hanging thread: IPC Reader 8 on port 49518 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84) org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.doRunLoop(HBaseServer.java:582) org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:568) java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) java.lang.Thread.run(Thread.java:662) Potentially hanging thread: IPC Server handler 1 on 49518 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399) org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1846) Potentially hanging thread: IPC Client (47) connection to asf001.sp2.ygridcore.net/67.195.138.31:60884 from jenkins.hfs.2 java.lang.Object.wait(Native Method) org.apache.hadoop.hbase.ipc.HBaseClient$Connection.waitForWork(HBaseClient.java:614) org.apache.hadoop.hbase.ipc.HBaseClient$Connection.run(HBaseClient.java:660) Potentially hanging thread: IPC Reader 5 on port 49518 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84) org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.doRunLoop(HBaseServer.java:582) org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:568) java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) java.lang.Thread.run(Thread.java:662) Potentially hanging thread: IPC Client (47) connection to asf001.sp2.ygridcore.net/67.195.138.31:49518 from jenkins java.lang.Object.wait(Native Method) org.apache.hadoop.hbase.ipc.HBaseClient$Connection.waitForWork(HBaseClient.java:614) org.apache.hadoop.hbase.ipc.HBaseClient$Connection.run(HBaseClient.java:660) Potentially hanging thread: RS_CLOSE_REGION-asf001.sp2.ygridcore.net,60884,1365336878389-1 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907) java.lang.Thread.run(Thread.java:662) Potentially hanging thread: hbase-am-zkevent-worker-pool-20-thread-29 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:424) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907) java.lang.Thread.run(Thread.java:662) Potentially hanging thread: IPC Server handler 4 on 49518 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399) org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1846) Potentially hanging thread: IPC Reader 3 on port 49518 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84) org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.doRunLoop(HBaseServer.java:582) org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:568) java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) java.lang.Thread.run(Thread.java:662) Potentially hanging thread: hbase-am-zkevent-worker-pool-2-thread-29 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:424) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907) java.lang.Thread.run(Thread.java:662) Potentially hanging thread: hbase-am-zkevent-worker-pool-2-thread-26 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:424) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907) java.lang.Thread.run(Thread.java:662) Potentially hanging thread: IPC Reader 4 on port 49518 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84) org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.doRunLoop(HBaseServer.java:582) org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:568) java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) java.lang.Thread.run(Thread.java:662) Potentially hanging thread: RegionServer:3;asf001.sp2.ygridcore.net,49518,1365336962323-SendThread(localhost:51597) sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80) org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:338) org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068) Potentially hanging thread: IPC Client (47) connection to asf001.sp2.ygridcore.net/67.195.138.31:43802 from jenkins java.lang.Object.wait(Native Method) org.apache.hadoop.hbase.ipc.HBaseClient$Connection.waitForWork(HBaseClient.java:614) org.apache.hadoop.hbase.ipc.HBaseClient$Connection.run(HBaseClient.java:660) Potentially hanging thread: IPC Client (47) connection to localhost/127.0.0.1:60405 from jenkins.hfs.2 java.lang.Object.wait(Native Method) org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:743) org.apache.hadoop.ipc.Client$Connection.run(Client.java:785) Potentially hanging thread: IPC Server handler 0 on 49518 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399) org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1846) Potentially hanging thread: IPC Client (47) connection to localhost/127.0.0.1:60405 from jenkins.hfs.3 java.lang.Object.wait(Native Method) org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:743) org.apache.hadoop.ipc.Client$Connection.run(Client.java:785) Potentially hanging thread: REPL IPC Server handler 1 on 49518 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399) org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1846) Potentially hanging thread: RegionServer:3;asf001.sp2.ygridcore.net,49518,1365336962323.logRoller java.lang.Object.wait(Native Method) org.apache.hadoop.hbase.regionserver.LogRoller.run(LogRoller.java:78) java.lang.Thread.run(Thread.java:662) Potentially hanging thread: PRI IPC Server handler 4 on 49518 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399) org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1846) Potentially hanging thread: hbase-am-zkevent-worker-pool-2-thread-24 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:424) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907) java.lang.Thread.run(Thread.java:662) Potentially hanging thread: PRI IPC Server handler 8 on 49518 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399) org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1846) Potentially hanging thread: PRI IPC Server handler 3 on 49518 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399) org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1846) Potentially hanging thread: IPC Client (47) connection to asf001.sp2.ygridcore.net/67.195.138.31:33796 from jenkins.hfs.3 java.lang.Object.wait(Native Method) org.apache.hadoop.hbase.ipc.HBaseClient$Connection.waitForWork(HBaseClient.java:614) org.apache.hadoop.hbase.ipc.HBaseClient$Connection.run(HBaseClient.java:660) Potentially hanging thread: RegionServer:3;asf001.sp2.ygridcore.net,49518,1365336962323-EventThread sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399) org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:491) Potentially hanging thread: PRI IPC Server handler 2 on 49518 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:158) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399) org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1846) Potentially hanging thread: RegionServer:3;asf001.sp2.ygridcore.net,49518,1365336962323-SendThread(localhost:51597) sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80) org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:338) org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068) Potentially hanging thread: hbase-am-zkevent-worker-pool-20-thread-28 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:424) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907) java.lang.Thread.run(Thread.java:662) Potentially hanging thread: hbase-am-zkevent-worker-pool-20-thread-26 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:424) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907) java.lang.Thread.run(Thread.java:662) Potentially hanging thread: hbase-am-zkevent-worker-pool-2-thread-27 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:424) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907) java.lang.Thread.run(Thread.java:662) Potentially hanging thread: hbase-am-zkevent-worker-pool-20-thread-32 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:424) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907) java.lang.Thread.run(Thread.java:662) Potentially hanging thread: hbase-am-zkevent-worker-pool-20-thread-36 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:424) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907) java.lang.Thread.run(Thread.java:662) Potentially hanging thread: RegionServer:3;asf001.sp2.ygridcore.net,49518,1365336962323.logSyncer java.lang.Object.wait(Native Method) org.apache.hadoop.hbase.regionserver.wal.FSHLog$LogSyncer.run(FSHLog.java:1015) java.lang.Thread.run(Thread.java:662) Potentially hanging thread: hbase-am-zkevent-worker-pool-2-thread-30 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:424) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907) java.lang.Thread.run(Thread.java:662) Potentially hanging thread: hbase-am-zkevent-worker-pool-20-thread-31 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:424) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907) java.lang.Thread.run(Thread.java:662) Potentially hanging thread: hbase-am-zkevent-worker-pool-2-thread-37 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:424) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907) java.lang.Thread.run(Thread.java:662) Potentially hanging thread: hbase-am-zkevent-worker-pool-20-thread-33 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:424) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:945) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907) java.lang.Thread.run(Thread.java:662) Potentially hanging thread: IPC Reader 9 on port 49518 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84) org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.doRunLoop(HBaseServer.java:582) org.apache.hadoop.hbase.ipc.HBaseServer$Listener$Reader.run(HBaseServer.java:568) java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) java.lang.Thread.run(Thread.java:662) - Thread LEAK? -, OpenFileDescriptor=505 (was 359) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=306 (was 306), ProcessCount=175 (was 175), AvailableMemoryMB=7169 (was 7197), ConnectionCount=6 (was 5) - ConnectionCount LEAK? - 2013-04-07 12:16:03,582 INFO [pool-1-thread-1] hbase.HBaseTestingUtility(888): Shutting down minicluster 2013-04-07 12:16:03,582 DEBUG [pool-1-thread-1] util.JVMClusterUtil(237): Shutting down HBase Cluster 2013-04-07 12:16:03,582 INFO [pool-1-thread-1] master.HMaster(2151): Cluster shutdown requested 2013-04-07 12:16:03,582 INFO [asf001.sp2.ygridcore.net,33796,1365336878036-BalancerChore] hbase.Chore(94): asf001.sp2.ygridcore.net,33796,1365336878036-BalancerChore exiting 2013-04-07 12:16:03,583 INFO [asf001.sp2.ygridcore.net,33796,1365336878036-CatalogJanitor] hbase.Chore(94): asf001.sp2.ygridcore.net,33796,1365336878036-CatalogJanitor exiting 2013-04-07 12:16:03,582 INFO [Master:0;asf001.sp2.ygridcore.net,33796,1365336878036] master.ServerManager(432): Waiting on regionserver(s) to go down asf001.sp2.ygridcore.net,43802,1365336889418, asf001.sp2.ygridcore.net,60884,1365336878389, asf001.sp2.ygridcore.net,49518,1365336962323 2013-04-07 12:16:03,582 INFO [asf001.sp2.ygridcore.net,33796,1365336878036-ClusterStatusChore] hbase.Chore(94): asf001.sp2.ygridcore.net,33796,1365336878036-ClusterStatusChore exiting 2013-04-07 12:16:03,584 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(274): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e-0x13de46b5f8b0015 Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2013-04-07 12:16:03,584 DEBUG [RegionServer:3;asf001.sp2.ygridcore.net,49518,1365336962323-EventThread] zookeeper.ZooKeeperWatcher(274): regionserver:49518-0x13de46b5f8b0012 Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2013-04-07 12:16:03,584 DEBUG [RegionServer:0;asf001.sp2.ygridcore.net,60884,1365336878389-EventThread] zookeeper.ZooKeeperWatcher(274): regionserver:60884-0x13de46b5f8b0001 Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2013-04-07 12:16:03,586 DEBUG [RegionServer:3;asf001.sp2.ygridcore.net,49518,1365336962323-EventThread] zookeeper.ZKUtil(424): regionserver:49518-0x13de46b5f8b0012 /hbase/running does not exist. Watcher is set. 2013-04-07 12:16:03,587 INFO [RegionServer:0;asf001.sp2.ygridcore.net,60884,1365336878389] regionserver.SplitLogWorker(519): Sending interrupt to stop the worker thread 2013-04-07 12:16:03,588 INFO [pool-1-thread-1] regionserver.HRegionServer(1545): STOPPED: Shutdown requested 2013-04-07 12:16:03,588 INFO [RegionServer:0;asf001.sp2.ygridcore.net,60884,1365336878389.logRoller] regionserver.LogRoller(119): LogRoller exiting. 2013-04-07 12:16:03,588 INFO [RegionServer:0;asf001.sp2.ygridcore.net,60884,1365336878389.compactionChecker] hbase.Chore(94): RegionServer:0;asf001.sp2.ygridcore.net,60884,1365336878389.compactionChecker exiting 2013-04-07 12:16:03,590 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZKUtil(424): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e-0x13de46b5f8b0015 /hbase/running does not exist. Watcher is set. 2013-04-07 12:16:03,588 INFO [RS_OPEN_META-asf001.sp2.ygridcore.net,60884,1365336878389-0MetaLogRoller] regionserver.LogRoller(119): LogRoller exiting. 2013-04-07 12:16:03,588 INFO [RegionServer:0;asf001.sp2.ygridcore.net,60884,1365336878389] snapshot.RegionServerSnapshotManager(151): Stopping RegionServerSnapshotManager gracefully. 2013-04-07 12:16:03,588 INFO [Thread-124] regionserver.MemStoreFlusher$FlushHandler(267): Thread-124 exiting 2013-04-07 12:16:03,591 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZKUtil(424): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e-0x13de46b5f8b0015 /hbase/running does not exist. Watcher is set. 2013-04-07 12:16:03,592 INFO [RegionServer:3;asf001.sp2.ygridcore.net,49518,1365336962323] regionserver.SplitLogWorker(519): Sending interrupt to stop the worker thread 2013-04-07 12:16:03,591 DEBUG [RS_CLOSE_REGION-asf001.sp2.ygridcore.net,60884,1365336878389-0] handler.CloseRegionHandler(125): Processing close of test1365336881340,,1365336881424.c9e9e308a02801bb891879b6cf471381. 2013-04-07 12:16:03,594 INFO [RegionServer:3;asf001.sp2.ygridcore.net,49518,1365336962323.compactionChecker] hbase.Chore(94): RegionServer:3;asf001.sp2.ygridcore.net,49518,1365336962323.compactionChecker exiting 2013-04-07 12:16:03,594 INFO [RegionServer:3;asf001.sp2.ygridcore.net,49518,1365336962323] snapshot.RegionServerSnapshotManager(151): Stopping RegionServerSnapshotManager gracefully. 2013-04-07 12:16:03,594 INFO [RegionServer:3;asf001.sp2.ygridcore.net,49518,1365336962323.logRoller] regionserver.LogRoller(119): LogRoller exiting. 2013-04-07 12:16:03,594 INFO [Thread-303] regionserver.MemStoreFlusher$FlushHandler(267): Thread-303 exiting 2013-04-07 12:16:03,595 INFO [StoreCloserThread-testLogSplittingAfterMasterRecoveryDueToZKExpiry,5,1365336889784.5403a557a6168e5f3c5141eb494f6088.-1] regionserver.HStore(622): Closed col 2013-04-07 12:16:03,594 INFO [StoreCloserThread-testLogSplittingAfterMasterRecoveryDueToZKExpiry,3,1365336889784.4e9a3d63237d29e6bf55bf2360834cbe.-1] regionserver.HStore(622): Closed col 2013-04-07 12:16:03,596 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZKUtil(424): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e-0x13de46b5f8b0015 /hbase/running does not exist. Watcher is set. 2013-04-07 12:16:03,594 INFO [SplitLogWorker-asf001.sp2.ygridcore.net,49518,1365336962323] regionserver.SplitLogWorker(205): SplitLogWorker interrupted while waiting for task, exiting: java.lang.InterruptedException 2013-04-07 12:16:03,596 INFO [SplitLogWorker-asf001.sp2.ygridcore.net,49518,1365336962323] regionserver.SplitLogWorker(169): SplitLogWorker asf001.sp2.ygridcore.net,49518,1365336962323 exiting 2013-04-07 12:16:03,596 DEBUG [RS_CLOSE_REGION-asf001.sp2.ygridcore.net,49518,1365336962323-2] handler.CloseRegionHandler(125): Processing close of testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,d,1365336962398.e99798d2c4d566ddb063efe03842457f. 2013-04-07 12:16:03,595 DEBUG [RS_CLOSE_REGION-asf001.sp2.ygridcore.net,49518,1365336962323-0] handler.CloseRegionHandler(125): Processing close of testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,g,1365336962398.682475e1358b7306bd3406e9330263a4. 2013-04-07 12:16:03,599 DEBUG [RS_CLOSE_REGION-asf001.sp2.ygridcore.net,49518,1365336962323-2] regionserver.HRegion(889): Closing testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,d,1365336962398.e99798d2c4d566ddb063efe03842457f.: disabling compactions & flushes 2013-04-07 12:16:03,600 DEBUG [RS_CLOSE_REGION-asf001.sp2.ygridcore.net,49518,1365336962323-0] regionserver.HRegion(889): Closing testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,g,1365336962398.682475e1358b7306bd3406e9330263a4.: disabling compactions & flushes 2013-04-07 12:16:03,600 DEBUG [RS_CLOSE_REGION-asf001.sp2.ygridcore.net,49518,1365336962323-2] regionserver.HRegion(911): Updates disabled for region testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,d,1365336962398.e99798d2c4d566ddb063efe03842457f. 2013-04-07 12:16:03,600 DEBUG [RS_CLOSE_REGION-asf001.sp2.ygridcore.net,49518,1365336962323-0] regionserver.HRegion(911): Updates disabled for region testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,g,1365336962398.682475e1358b7306bd3406e9330263a4. 2013-04-07 12:16:03,600 INFO [RS_CLOSE_REGION-asf001.sp2.ygridcore.net,49518,1365336962323-2] regionserver.HRegion(968): Closed testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,d,1365336962398.e99798d2c4d566ddb063efe03842457f. 2013-04-07 12:16:03,600 INFO [RS_CLOSE_REGION-asf001.sp2.ygridcore.net,49518,1365336962323-0] regionserver.HRegion(968): Closed testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,g,1365336962398.682475e1358b7306bd3406e9330263a4. 2013-04-07 12:16:03,600 DEBUG [RS_CLOSE_REGION-asf001.sp2.ygridcore.net,49518,1365336962323-2] handler.CloseRegionHandler(177): Closed region testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,d,1365336962398.e99798d2c4d566ddb063efe03842457f. 2013-04-07 12:16:03,600 DEBUG [RS_CLOSE_REGION-asf001.sp2.ygridcore.net,49518,1365336962323-0] handler.CloseRegionHandler(177): Closed region testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,g,1365336962398.682475e1358b7306bd3406e9330263a4. 2013-04-07 12:16:03,595 INFO [RegionServer:3;asf001.sp2.ygridcore.net,49518,1365336962323] regionserver.HRegionServer(833): stopping server asf001.sp2.ygridcore.net,49518,1365336962323 2013-04-07 12:16:03,595 DEBUG [RS_CLOSE_REGION-asf001.sp2.ygridcore.net,60884,1365336878389-0] regionserver.HRegion(889): Closing test1365336881340,,1365336881424.c9e9e308a02801bb891879b6cf471381.: disabling compactions & flushes 2013-04-07 12:16:03,601 DEBUG [RegionServer:3;asf001.sp2.ygridcore.net,49518,1365336962323] catalog.CatalogTracker(208): Stopping catalog tracker org.apache.hadoop.hbase.catalog.CatalogTracker@1945696 2013-04-07 12:16:03,601 DEBUG [RS_CLOSE_REGION-asf001.sp2.ygridcore.net,60884,1365336878389-0] regionserver.HRegion(911): Updates disabled for region test1365336881340,,1365336881424.c9e9e308a02801bb891879b6cf471381. 2013-04-07 12:16:03,599 DEBUG [RS_CLOSE_REGION-asf001.sp2.ygridcore.net,60884,1365336878389-1] regionserver.HRegion(889): Closing testRegionAssignmentAfterMasterRecoveryDueToZKExpiry,i,1365336962398.385e5846630b0cb4493758961a2c2457.: disabling compactions & flushes 2013-04-07 12:16:03,601 DEBUG [RS_CLOSE_REGION-asf001.sp2.ygridcore.net,60884,1365336878389-0] regionserver.HRegion(1391): Started memstore flush for test1365336881340,,1365336881424.c9e9e308a02801bb891879b6cf471381., current region memstore size 136.0 2013-04-07 12:16:03,698 INFO [RegionServer:3;asf001.sp2.ygridcore.net,49518,1365336962323] client.HConnectionManager$HConnectionImplementation(1672): Closing zookeeper sessionid=0x13de46b5f8b0013 2013-04-07 12:16:03,801 INFO [RegionServer:3;asf001.sp2.ygridcore.net,49518,1365336962323] snapshot.RegionServerSnapshotManager(151): Stopping RegionServerSnapshotManager gracefully. 2013-04-07 12:16:03,801 INFO [RegionServer:0;asf001.sp2.ygridcore.net,60884,1365336878389] client.HConnectionManager$HConnectionImplementation(1672): Closing zookeeper sessionid=0x13de46b5f8b0005 2013-04-07 12:16:03,801 INFO [RegionServer:3;asf001.sp2.ygridcore.net,49518,1365336962323] regionserver.HRegionServer(859): stopping server asf001.sp2.ygridcore.net,49518,1365336962323; all regions closed. 2013-04-07 12:16:03,802 INFO [RegionServer:3;asf001.sp2.ygridcore.net,49518,1365336962323.logSyncer] wal.FSHLog$LogSyncer(1030): RegionServer:3;asf001.sp2.ygridcore.net,49518,1365336962323.logSyncer exiting 2013-04-07 12:16:03,802 DEBUG [RegionServer:3;asf001.sp2.ygridcore.net,49518,1365336962323] wal.FSHLog(836): closing hlog writer in hdfs://localhost:60405/user/jenkins/hbase/.logs/asf001.sp2.ygridcore.net,49518,1365336962323 2013-04-07 12:16:03,814 DEBUG [RegionServer:3;asf001.sp2.ygridcore.net,49518,1365336962323] wal.FSHLog(796): Moved 1 log files to /user/jenkins/hbase/.oldlogs 2013-04-07 12:16:03,830 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(274): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e-0x13de46b5f8b0015 Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/asf001.sp2.ygridcore.net,43802,1365336889418 2013-04-07 12:16:03,830 INFO [pool-1-thread-1-EventThread] zookeeper.RegionServerTracker(94): RegionServer ephemeral node deleted, processing expiration [asf001.sp2.ygridcore.net,43802,1365336889418] 2013-04-07 12:16:03,830 INFO [pool-1-thread-1-EventThread] master.ServerManager(479): Cluster shutdown set; asf001.sp2.ygridcore.net,43802,1365336889418 expired; onlineServers=2 2013-04-07 12:16:03,830 INFO [pool-1-thread-1-EventThread] zookeeper.RegionServerTracker(94): RegionServer ephemeral node deleted, processing expiration [asf001.sp2.ygridcore.net,43802,1365336889418] 2013-04-07 12:16:03,830 WARN [pool-1-thread-1-EventThread] zookeeper.RegionServerTracker(98): asf001.sp2.ygridcore.net,43802,1365336889418 is not online or isn't known to the master.The latter could be caused by a DNS misconfiguration. 2013-04-07 12:16:03,830 INFO [pool-1-thread-1-EventThread] zookeeper.RegionServerTracker(94): RegionServer ephemeral node deleted, processing expiration [asf001.sp2.ygridcore.net,43802,1365336889418] 2013-04-07 12:16:03,830 WARN [pool-1-thread-1-EventThread] zookeeper.RegionServerTracker(98): asf001.sp2.ygridcore.net,43802,1365336889418 is not online or isn't known to the master.The latter could be caused by a DNS misconfiguration. 2013-04-07 12:16:03,830 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(274): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e-0x13de46b5f8b0015 Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2013-04-07 12:16:03,833 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZKUtil(422): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e-0x13de46b5f8b0015 Set watcher on existing znode /hbase/rs/asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:16:03,834 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZKUtil(422): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e-0x13de46b5f8b0015 Set watcher on existing znode /hbase/rs/asf001.sp2.ygridcore.net,49518,1365336962323 2013-04-07 12:16:03,835 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZKUtil(422): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e-0x13de46b5f8b0015 Set watcher on existing znode /hbase/rs/asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:16:03,836 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZKUtil(422): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e-0x13de46b5f8b0015 Set watcher on existing znode /hbase/rs/asf001.sp2.ygridcore.net,49518,1365336962323 2013-04-07 12:16:03,837 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZKUtil(422): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e-0x13de46b5f8b0015 Set watcher on existing znode /hbase/rs/asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:16:03,837 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZKUtil(422): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e-0x13de46b5f8b0015 Set watcher on existing znode /hbase/rs/asf001.sp2.ygridcore.net,49518,1365336962323 2013-04-07 12:16:03,903 INFO [RegionServer:0;asf001.sp2.ygridcore.net,60884,1365336878389] snapshot.RegionServerSnapshotManager(151): Stopping RegionServerSnapshotManager gracefully. 2013-04-07 12:16:03,903 INFO [RegionServer:0;asf001.sp2.ygridcore.net,60884,1365336878389] regionserver.CompactSplitThread(318): Waiting for Split Thread to finish... 2013-04-07 12:16:03,904 INFO [RegionServer:0;asf001.sp2.ygridcore.net,60884,1365336878389] regionserver.CompactSplitThread(318): Waiting for Merge Thread to finish... 2013-04-07 12:16:03,904 INFO [RegionServer:0;asf001.sp2.ygridcore.net,60884,1365336878389] regionserver.CompactSplitThread(318): Waiting for Large Compaction Thread to finish... 2013-04-07 12:16:03,904 INFO [RegionServer:0;asf001.sp2.ygridcore.net,60884,1365336878389] regionserver.CompactSplitThread(318): Waiting for Small Compaction Thread to finish... 2013-04-07 12:16:03,904 INFO [RegionServer:0;asf001.sp2.ygridcore.net,60884,1365336878389] regionserver.HRegionServer(993): Waiting on 1 regions to close 2013-04-07 12:16:03,904 DEBUG [RS_CLOSE_META-asf001.sp2.ygridcore.net,60884,1365336878389-0] handler.CloseRegionHandler(125): Processing close of .META.,,1.1028785192 2013-04-07 12:16:03,904 DEBUG [RegionServer:0;asf001.sp2.ygridcore.net,60884,1365336878389] regionserver.HRegionServer(997): {1028785192=.META.,,1.1028785192} 2013-04-07 12:16:03,905 DEBUG [RS_CLOSE_META-asf001.sp2.ygridcore.net,60884,1365336878389-0] regionserver.HRegion(889): Closing .META.,,1.1028785192: disabling compactions & flushes 2013-04-07 12:16:03,905 DEBUG [RS_CLOSE_META-asf001.sp2.ygridcore.net,60884,1365336878389-0] regionserver.HRegion(911): Updates disabled for region .META.,,1.1028785192 2013-04-07 12:16:03,905 DEBUG [RS_CLOSE_META-asf001.sp2.ygridcore.net,60884,1365336878389-0] regionserver.HRegion(1391): Started memstore flush for .META.,,1.1028785192, current region memstore size 18.9k 2013-04-07 12:16:03,905 DEBUG [RS_CLOSE_META-asf001.sp2.ygridcore.net,60884,1365336878389-0] regionserver.HRegion(1447): Finished snapshotting .META.,,1.1028785192, commencing wait for mvcc, flushsize=19304 2013-04-07 12:16:03,905 DEBUG [RS_CLOSE_META-asf001.sp2.ygridcore.net,60884,1365336878389-0] regionserver.HRegion(1463): Finished snapshotting, commencing flushing stores 2013-04-07 12:16:03,909 DEBUG [RS_CLOSE_META-asf001.sp2.ygridcore.net,60884,1365336878389-0] util.FSUtils(211): Creating file=hdfs://localhost:60405/user/jenkins/hbase/.META./1028785192/.tmp/06f8b8d97e4c412998b8630a2eac54e6 with permission=rwxrwxrwx 2013-04-07 12:16:03,911 DEBUG [RS_CLOSE_META-asf001.sp2.ygridcore.net,60884,1365336878389-0] hfile.HFileWriterV2(146): Initialized with CacheConfig:enabled [cacheDataOnRead=true] [cacheDataOnWrite=false] [cacheIndexesOnWrite=false] [cacheBloomsOnWrite=false] [cacheEvictOnClose=false] [cacheCompressed=false] 2013-04-07 12:16:03,912 INFO [RS_CLOSE_META-asf001.sp2.ygridcore.net,60884,1365336878389-0] regionserver.StoreFile$Writer(813): Delete Family Bloom filter type for hdfs://localhost:60405/user/jenkins/hbase/.META./1028785192/.tmp/06f8b8d97e4c412998b8630a2eac54e6: CompoundBloomFilterWriter 2013-04-07 12:16:03,916 INFO [RegionServer:3;asf001.sp2.ygridcore.net,49518,1365336962323] regionserver.Leases(124): RegionServer:3;asf001.sp2.ygridcore.net,49518,1365336962323 closing leases 2013-04-07 12:16:03,916 INFO [RegionServer:3;asf001.sp2.ygridcore.net,49518,1365336962323] regionserver.Leases(131): RegionServer:3;asf001.sp2.ygridcore.net,49518,1365336962323 closed leases 2013-04-07 12:16:03,916 INFO [RegionServer:3;asf001.sp2.ygridcore.net,49518,1365336962323] regionserver.CompactSplitThread(318): Waiting for Split Thread to finish... 2013-04-07 12:16:03,916 INFO [RegionServer:3;asf001.sp2.ygridcore.net,49518,1365336962323] regionserver.CompactSplitThread(318): Waiting for Merge Thread to finish... 2013-04-07 12:16:03,916 INFO [RegionServer:3;asf001.sp2.ygridcore.net,49518,1365336962323] regionserver.CompactSplitThread(318): Waiting for Large Compaction Thread to finish... 2013-04-07 12:16:03,916 INFO [RegionServer:3;asf001.sp2.ygridcore.net,49518,1365336962323] regionserver.CompactSplitThread(318): Waiting for Small Compaction Thread to finish... 2013-04-07 12:16:03,919 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(274): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e-0x13de46b5f8b0015 Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/asf001.sp2.ygridcore.net,49518,1365336962323 2013-04-07 12:16:03,919 INFO [pool-1-thread-1-EventThread] zookeeper.RegionServerTracker(94): RegionServer ephemeral node deleted, processing expiration [asf001.sp2.ygridcore.net,49518,1365336962323] 2013-04-07 12:16:03,919 INFO [pool-1-thread-1-EventThread] master.ServerManager(479): Cluster shutdown set; asf001.sp2.ygridcore.net,49518,1365336962323 expired; onlineServers=1 2013-04-07 12:16:03,919 INFO [pool-1-thread-1-EventThread] zookeeper.RegionServerTracker(94): RegionServer ephemeral node deleted, processing expiration [asf001.sp2.ygridcore.net,49518,1365336962323] 2013-04-07 12:16:03,919 WARN [pool-1-thread-1-EventThread] zookeeper.RegionServerTracker(98): asf001.sp2.ygridcore.net,49518,1365336962323 is not online or isn't known to the master.The latter could be caused by a DNS misconfiguration. 2013-04-07 12:16:03,920 INFO [pool-1-thread-1-EventThread] zookeeper.RegionServerTracker(94): RegionServer ephemeral node deleted, processing expiration [asf001.sp2.ygridcore.net,49518,1365336962323] 2013-04-07 12:16:03,920 WARN [pool-1-thread-1-EventThread] zookeeper.RegionServerTracker(98): asf001.sp2.ygridcore.net,49518,1365336962323 is not online or isn't known to the master.The latter could be caused by a DNS misconfiguration. 2013-04-07 12:16:03,920 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(274): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e-0x13de46b5f8b0015 Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2013-04-07 12:16:03,924 INFO [RegionServer:3;asf001.sp2.ygridcore.net,49518,1365336962323] regionserver.HRegionServer(888): stopping server asf001.sp2.ygridcore.net,49518,1365336962323; zookeeper connection closed. 2013-04-07 12:16:03,924 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZKUtil(422): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e-0x13de46b5f8b0015 Set watcher on existing znode /hbase/rs/asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:16:03,924 INFO [RegionServer:3;asf001.sp2.ygridcore.net,49518,1365336962323] regionserver.HRegionServer(891): RegionServer:3;asf001.sp2.ygridcore.net,49518,1365336962323 exiting 2013-04-07 12:16:03,926 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZKUtil(422): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e-0x13de46b5f8b0015 Set watcher on existing znode /hbase/rs/asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:16:03,927 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@123b9c1] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(186): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@123b9c1 2013-04-07 12:16:03,927 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZKUtil(422): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e-0x13de46b5f8b0015 Set watcher on existing znode /hbase/rs/asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:16:04,071 INFO [RegionServer:0;asf001.sp2.ygridcore.net,60884,1365336878389.leaseChecker] regionserver.Leases(124): RegionServer:0;asf001.sp2.ygridcore.net,60884,1365336878389.leaseChecker closing leases 2013-04-07 12:16:04,071 INFO [RegionServer:0;asf001.sp2.ygridcore.net,60884,1365336878389.leaseChecker] regionserver.Leases(131): RegionServer:0;asf001.sp2.ygridcore.net,60884,1365336878389.leaseChecker closed leases 2013-04-07 12:16:04,335 INFO [RS_CLOSE_META-asf001.sp2.ygridcore.net,60884,1365336878389-0] regionserver.StoreFile$Writer(1033): NO General Bloom and NO DeleteFamily was added to HFile (hdfs://localhost:60405/user/jenkins/hbase/.META./1028785192/.tmp/06f8b8d97e4c412998b8630a2eac54e6) 2013-04-07 12:16:04,335 INFO [RS_CLOSE_META-asf001.sp2.ygridcore.net,60884,1365336878389-0] regionserver.HStore(791): Flushed , sequenceid=25, memsize=18.9k, into tmp file hdfs://localhost:60405/user/jenkins/hbase/.META./1028785192/.tmp/06f8b8d97e4c412998b8630a2eac54e6 2013-04-07 12:16:04,342 DEBUG [RS_CLOSE_META-asf001.sp2.ygridcore.net,60884,1365336878389-0] regionserver.HRegionFileSystem(338): Committing store file hdfs://localhost:60405/user/jenkins/hbase/.META./1028785192/.tmp/06f8b8d97e4c412998b8630a2eac54e6 as hdfs://localhost:60405/user/jenkins/hbase/.META./1028785192/info/06f8b8d97e4c412998b8630a2eac54e6 2013-04-07 12:16:04,349 INFO [RS_CLOSE_META-asf001.sp2.ygridcore.net,60884,1365336878389-0] regionserver.HStore(823): Added hdfs://localhost:60405/user/jenkins/hbase/.META./1028785192/info/06f8b8d97e4c412998b8630a2eac54e6, entries=76, sequenceid=25, filesize=13.1k 2013-04-07 12:16:04,349 INFO [RS_CLOSE_META-asf001.sp2.ygridcore.net,60884,1365336878389-0] regionserver.HRegion(1534): Finished memstore flush of ~18.9k/19304, currentsize=0.0/0 for region .META.,,1.1028785192 in 444ms, sequenceid=25, compaction requested=false 2013-04-07 12:16:04,350 INFO [StoreCloserThread-.META.,,1.1028785192-1] regionserver.HStore(622): Closed info 2013-04-07 12:16:04,351 INFO [RS_CLOSE_META-asf001.sp2.ygridcore.net,60884,1365336878389-0] regionserver.HRegion(968): Closed .META.,,1.1028785192 2013-04-07 12:16:04,351 DEBUG [RS_CLOSE_META-asf001.sp2.ygridcore.net,60884,1365336878389-0] handler.CloseRegionHandler(177): Closed region .META.,,1.1028785192 2013-04-07 12:16:04,384 INFO [RegionServer:3;asf001.sp2.ygridcore.net,49518,1365336962323.leaseChecker] regionserver.Leases(124): RegionServer:3;asf001.sp2.ygridcore.net,49518,1365336962323.leaseChecker closing leases 2013-04-07 12:16:04,384 INFO [RegionServer:3;asf001.sp2.ygridcore.net,49518,1365336962323.leaseChecker] regionserver.Leases(131): RegionServer:3;asf001.sp2.ygridcore.net,49518,1365336962323.leaseChecker closed leases 2013-04-07 12:16:04,483 INFO [asf001.sp2.ygridcore.net,33796,1365336878036.splitLogManagerTimeoutMonitor] hbase.Chore(94): asf001.sp2.ygridcore.net,33796,1365336878036.splitLogManagerTimeoutMonitor exiting 2013-04-07 12:16:04,505 INFO [RegionServer:0;asf001.sp2.ygridcore.net,60884,1365336878389] regionserver.HRegionServer(859): stopping server asf001.sp2.ygridcore.net,60884,1365336878389; all regions closed. 2013-04-07 12:16:04,505 INFO [RS_OPEN_META-asf001.sp2.ygridcore.net,60884,1365336878389-0.logSyncer] wal.FSHLog$LogSyncer(1030): RS_OPEN_META-asf001.sp2.ygridcore.net,60884,1365336878389-0.logSyncer exiting 2013-04-07 12:16:04,506 DEBUG [RegionServer:0;asf001.sp2.ygridcore.net,60884,1365336878389] wal.FSHLog(836): closing hlog writer in hdfs://localhost:60405/user/jenkins/hbase/.logs/asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:16:04,620 INFO [Master:0;asf001.sp2.ygridcore.net,33796,1365336878036] master.ServerManager(432): Waiting on regionserver(s) to go down asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:16:04,921 INFO [RegionServer:0;asf001.sp2.ygridcore.net,60884,1365336878389.logSyncer] wal.FSHLog$LogSyncer(1030): RegionServer:0;asf001.sp2.ygridcore.net,60884,1365336878389.logSyncer exiting 2013-04-07 12:16:04,922 DEBUG [RegionServer:0;asf001.sp2.ygridcore.net,60884,1365336878389] wal.FSHLog(836): closing hlog writer in hdfs://localhost:60405/user/jenkins/hbase/.logs/asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:16:04,932 DEBUG [RegionServer:0;asf001.sp2.ygridcore.net,60884,1365336878389] wal.FSHLog(796): Moved 2 log files to /user/jenkins/hbase/.oldlogs 2013-04-07 12:16:05,034 INFO [RegionServer:0;asf001.sp2.ygridcore.net,60884,1365336878389] regionserver.Leases(124): RegionServer:0;asf001.sp2.ygridcore.net,60884,1365336878389 closing leases 2013-04-07 12:16:05,034 INFO [RegionServer:0;asf001.sp2.ygridcore.net,60884,1365336878389] regionserver.Leases(131): RegionServer:0;asf001.sp2.ygridcore.net,60884,1365336878389 closed leases 2013-04-07 12:16:05,037 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(274): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e-0x13de46b5f8b0015 Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/asf001.sp2.ygridcore.net,60884,1365336878389 2013-04-07 12:16:05,037 INFO [pool-1-thread-1-EventThread] zookeeper.RegionServerTracker(94): RegionServer ephemeral node deleted, processing expiration [asf001.sp2.ygridcore.net,60884,1365336878389] 2013-04-07 12:16:05,037 INFO [pool-1-thread-1-EventThread] master.ServerManager(479): Cluster shutdown set; asf001.sp2.ygridcore.net,60884,1365336878389 expired; onlineServers=0 2013-04-07 12:16:05,037 DEBUG [Master:0;asf001.sp2.ygridcore.net,33796,1365336878036] master.HMaster(1063): Stopping service threads 2013-04-07 12:16:05,037 INFO [pool-1-thread-1-EventThread] master.HMaster(2151): Cluster shutdown set; onlineServer=0 2013-04-07 12:16:05,039 INFO [pool-1-thread-1-EventThread] zookeeper.RegionServerTracker(94): RegionServer ephemeral node deleted, processing expiration [asf001.sp2.ygridcore.net,60884,1365336878389] 2013-04-07 12:16:05,040 WARN [pool-1-thread-1-EventThread] zookeeper.RegionServerTracker(98): asf001.sp2.ygridcore.net,60884,1365336878389 is not online or isn't known to the master.The latter could be caused by a DNS misconfiguration. 2013-04-07 12:16:05,040 INFO [Master:0;asf001.sp2.ygridcore.net,33796,1365336878036.archivedHFileCleaner] hbase.Chore(94): Master:0;asf001.sp2.ygridcore.net,33796,1365336878036.archivedHFileCleaner exiting 2013-04-07 12:16:05,040 INFO [pool-1-thread-1-EventThread] zookeeper.RegionServerTracker(94): RegionServer ephemeral node deleted, processing expiration [asf001.sp2.ygridcore.net,60884,1365336878389] 2013-04-07 12:16:05,040 INFO [Master:0;asf001.sp2.ygridcore.net,33796,1365336878036.oldLogCleaner] hbase.Chore(94): Master:0;asf001.sp2.ygridcore.net,33796,1365336878036.oldLogCleaner exiting 2013-04-07 12:16:05,040 WARN [pool-1-thread-1-EventThread] zookeeper.RegionServerTracker(98): asf001.sp2.ygridcore.net,60884,1365336878389 is not online or isn't known to the master.The latter could be caused by a DNS misconfiguration. 2013-04-07 12:16:05,041 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(274): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e-0x13de46b5f8b0015 Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2013-04-07 12:16:05,041 INFO [RegionServer:0;asf001.sp2.ygridcore.net,60884,1365336878389] regionserver.HRegionServer(888): stopping server asf001.sp2.ygridcore.net,60884,1365336878389; zookeeper connection closed. 2013-04-07 12:16:05,041 INFO [RegionServer:0;asf001.sp2.ygridcore.net,60884,1365336878389] regionserver.HRegionServer(891): RegionServer:0;asf001.sp2.ygridcore.net,60884,1365336878389 exiting 2013-04-07 12:16:05,042 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@cef147] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(186): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@cef147 2013-04-07 12:16:05,042 INFO [pool-1-thread-1] util.JVMClusterUtil(309): Shutdown of 1 master(s) and 4 regionserver(s) complete 2013-04-07 12:16:05,042 DEBUG [Master:0;asf001.sp2.ygridcore.net,33796,1365336878036] zookeeper.ZKUtil(1680): master:33796-0x13de46b5f8b0000-0x13de46b5f8b000e-0x13de46b5f8b0015 Retrieved 43 byte(s) of data from znode /hbase/master; data=PBUF\x0A%\x0A\x18asf001.sp2.y... 2013-04-07 12:16:05,042 INFO [pool-1-thread-1] client.HConnectionManager$HConnectionImplementation(1672): Closing zookeeper sessionid=0x13de46b5f8b000d 2013-04-07 12:16:05,045 DEBUG [Master:0;asf001.sp2.ygridcore.net,33796,1365336878036] catalog.CatalogTracker(208): Stopping catalog tracker org.apache.hadoop.hbase.catalog.CatalogTracker@f92541 2013-04-07 12:16:05,047 INFO [Master:0;asf001.sp2.ygridcore.net,33796,1365336878036] master.HMaster(557): HMaster main thread exiting 2013-04-07 12:16:05,050 INFO [pool-1-thread-1] zookeeper.MiniZooKeeperCluster(246): Shutdown MiniZK cluster with all ZK servers Shutting down the Mini HDFS Cluster Shutting down DataNode 1 2013-04-07 12:16:05,050 INFO [pool-1-thread-1] log.Slf4jLog(67): Stopped SelectChannelConnector@localhost:0 2013-04-07 12:16:05,148 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(274): unittest-0x13de46b5f8b0014 Received ZooKeeper Event, type=None, state=Disconnected, path=null 2013-04-07 12:16:05,148 DEBUG [pool-1-thread-1-EventThread] zookeeper.ZooKeeperWatcher(356): unittest-0x13de46b5f8b0014 Received Disconnected from ZooKeeper, ignoring