2016-01-11 22:48:19,815 INFO [main] hbase.ResourceChecker(147): before: regionserver.TestSplitWalDataLoss#test Thread=4, OpenFileDescriptor=166, MaxFileDescriptor=65535, SystemLoadAverage=312, ProcessCount=561, AvailableMemoryMB=9633, ConnectionCount=0 2016-01-11 22:48:19,819 INFO [main] hbase.HBaseTestingUtility(967): Starting up minicluster with 1 master(s) and 2 regionserver(s) and 2 datanode(s) 2016-01-11 22:48:19,847 INFO [main] hbase.HBaseTestingUtility(447): Created new mini-cluster data directory: /data/users/elliott/hbase/hbase-server/target/test-data/106c9698-d5e9-426f-9886-d717fa502d54/dfscluster_3e875bf4-f4b9-453c-966d-153d9ade8758, deleteOnExit=true 2016-01-11 22:48:19,847 INFO [main] hbase.HBaseTestingUtility(697): Setting test.cache.data to /data/users/elliott/hbase/hbase-server/target/test-data/106c9698-d5e9-426f-9886-d717fa502d54/cache_data in system properties and HBase conf 2016-01-11 22:48:19,848 INFO [main] hbase.HBaseTestingUtility(697): Setting hadoop.tmp.dir to /data/users/elliott/hbase/hbase-server/target/test-data/106c9698-d5e9-426f-9886-d717fa502d54/hadoop_tmp in system properties and HBase conf 2016-01-11 22:48:19,848 INFO [main] hbase.HBaseTestingUtility(697): Setting hadoop.log.dir to /data/users/elliott/hbase/hbase-server/target/test-data/106c9698-d5e9-426f-9886-d717fa502d54/hadoop_logs in system properties and HBase conf 2016-01-11 22:48:19,849 INFO [main] hbase.HBaseTestingUtility(697): Setting mapreduce.cluster.local.dir to /data/users/elliott/hbase/hbase-server/target/test-data/106c9698-d5e9-426f-9886-d717fa502d54/mapred_local in system properties and HBase conf 2016-01-11 22:48:19,849 INFO [main] hbase.HBaseTestingUtility(697): Setting mapreduce.cluster.temp.dir to /data/users/elliott/hbase/hbase-server/target/test-data/106c9698-d5e9-426f-9886-d717fa502d54/mapred_temp in system properties and HBase conf 2016-01-11 22:48:19,849 INFO [main] hbase.HBaseTestingUtility(688): read short circuit is OFF 2016-01-11 22:48:19,957 WARN [main] util.NativeCodeLoader(62): Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 2016-01-11 22:48:20,103 DEBUG [main] fs.HFileSystem(221): The file system is not a DistributedFileSystem. Skipping on block location reordering Formatting using clusterid: testClusterID 2016-01-11 22:48:21,153 WARN [main] impl.MetricsConfig(124): Cannot locate configuration: tried hadoop-metrics2-namenode.properties,hadoop-metrics2.properties 2016-01-11 22:48:21,296 INFO [main] log.Slf4jLog(67): Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog 2016-01-11 22:48:21,397 INFO [main] log.Slf4jLog(67): jetty-6.1.26 2016-01-11 22:48:21,425 INFO [main] log.Slf4jLog(67): Extract jar:file:/home/elliott/.m2/repository/org/apache/hadoop/hadoop-hdfs/2.5.1/hadoop-hdfs-2.5.1-tests.jar!/webapps/hdfs to /tmp/Jetty_localhost_60806_hdfs____.r3dxqq/webapp 2016-01-11 22:48:21,569 WARN [main] server.AuthenticationFilter(166): 'signature.secret' configuration not set, using a random value as secret 2016-01-11 22:48:21,607 INFO [main] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:60806 2016-01-11 22:48:22,167 INFO [main] log.Slf4jLog(67): jetty-6.1.26 2016-01-11 22:48:22,171 INFO [main] log.Slf4jLog(67): Extract jar:file:/home/elliott/.m2/repository/org/apache/hadoop/hadoop-hdfs/2.5.1/hadoop-hdfs-2.5.1-tests.jar!/webapps/datanode to /tmp/Jetty_localhost_36258_datanode____.jmompz/webapp 2016-01-11 22:48:22,266 INFO [main] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:36258 2016-01-11 22:48:22,316 INFO [main] log.Slf4jLog(67): jetty-6.1.26 2016-01-11 22:48:22,319 INFO [main] log.Slf4jLog(67): Extract jar:file:/home/elliott/.m2/repository/org/apache/hadoop/hadoop-hdfs/2.5.1/hadoop-hdfs-2.5.1-tests.jar!/webapps/datanode to /tmp/Jetty_localhost_64917_datanode____.6wgtmg/webapp 2016-01-11 22:48:22,411 INFO [main] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:64917 2016-01-11 22:48:23,160 INFO [IPC Server handler 3 on 59295] blockmanagement.BlockManager(1753): BLOCK* processReport: from storage DS-2f4161bb-a789-4c3e-bb22-3dbf1babdf25 node DatanodeRegistration(127.0.0.1, datanodeUuid=83780db6-1977-4d15-a635-31f452ae5cf3, infoPort=36258, ipcPort=42627, storageInfo=lv=-55;cid=testClusterID;nsid=1088338929;c=0), blocks: 0, hasStaleStorages: true, processing time: 1 msecs 2016-01-11 22:48:23,161 INFO [IPC Server handler 3 on 59295] blockmanagement.BlockManager(1753): BLOCK* processReport: from storage DS-e3b160d5-0860-46e6-8012-66df73514c0f node DatanodeRegistration(127.0.0.1, datanodeUuid=83780db6-1977-4d15-a635-31f452ae5cf3, infoPort=36258, ipcPort=42627, storageInfo=lv=-55;cid=testClusterID;nsid=1088338929;c=0), blocks: 0, hasStaleStorages: false, processing time: 0 msecs 2016-01-11 22:48:23,181 INFO [IPC Server handler 4 on 59295] blockmanagement.BlockManager(1753): BLOCK* processReport: from storage DS-4449a44e-3e96-4db5-b41c-f22c79a44f57 node DatanodeRegistration(127.0.0.1, datanodeUuid=13b3e9de-5a2b-47a5-b897-0039ad33a71c, infoPort=64917, ipcPort=37012, storageInfo=lv=-55;cid=testClusterID;nsid=1088338929;c=0), blocks: 0, hasStaleStorages: true, processing time: 1 msecs 2016-01-11 22:48:23,181 INFO [IPC Server handler 4 on 59295] blockmanagement.BlockManager(1753): BLOCK* processReport: from storage DS-940d5c3a-1db1-4402-9e4c-a19c2cc6af02 node DatanodeRegistration(127.0.0.1, datanodeUuid=13b3e9de-5a2b-47a5-b897-0039ad33a71c, infoPort=64917, ipcPort=37012, storageInfo=lv=-55;cid=testClusterID;nsid=1088338929;c=0), blocks: 0, hasStaleStorages: false, processing time: 0 msecs 2016-01-11 22:48:23,325 INFO [main] zookeeper.MiniZooKeeperCluster(273): Started MiniZooKeeperCluster and ran successful 'stat' on client port=55120 2016-01-11 22:48:23,341 INFO [main] fs.HFileSystem(252): Added intercepting call to namenode#getBlockLocations so can do block reordering using class class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2016-01-11 22:48:23,343 INFO [main] fs.HFileSystem(252): Added intercepting call to namenode#getBlockLocations so can do block reordering using class class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2016-01-11 22:48:23,789 INFO [IPC Server handler 3 on 59295] blockmanagement.BlockManager(2383): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:38785 is added to blk_1073741825_1001{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-940d5c3a-1db1-4402-9e4c-a19c2cc6af02:NORMAL|RBW], ReplicaUnderConstruction[[DISK]DS-2f4161bb-a789-4c3e-bb22-3dbf1babdf25:NORMAL|RBW]]} size 0 2016-01-11 22:48:23,798 INFO [IPC Server handler 5 on 59295] blockmanagement.BlockManager(2383): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:46850 is added to blk_1073741825_1001 size 7 2016-01-11 22:48:23,809 INFO [main] util.FSUtils(747): Created version file at hdfs://localhost:59295/user/elliott/test-data/4f0ddc3b-9bf4-4ec6-b2ab-29e91003a033 with version=8 2016-01-11 22:48:23,977 INFO [main] client.ConnectionUtils(104): master/dev1762.prn2.facebook.com/10.35.11.29:0 server-side HConnection retries=350 2016-01-11 22:48:24,122 INFO [main] ipc.SimpleRpcScheduler(128): Using deadline as user call queue, count=1 2016-01-11 22:48:24,130 INFO [main] ipc.RpcServer$Listener(637): master/dev1762.prn2.facebook.com/10.35.11.29:0: started 3 reader(s) listening on port=49897 2016-01-11 22:48:24,161 INFO [main] fs.HFileSystem(252): Added intercepting call to namenode#getBlockLocations so can do block reordering using class class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2016-01-11 22:48:24,215 INFO [main] zookeeper.RecoverableZooKeeper(120): Process identifier=master:49897 connecting to ZooKeeper ensemble=localhost:55120 2016-01-11 22:48:24,298 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(554): master:498970x0, quorum=localhost:55120, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2016-01-11 22:48:24,301 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(638): master:49897-0x1523497b4040000 connected 2016-01-11 22:48:24,406 DEBUG [main] zookeeper.ZKUtil(370): master:49897-0x1523497b4040000, quorum=localhost:55120, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2016-01-11 22:48:24,407 DEBUG [main] zookeeper.ZKUtil(370): master:49897-0x1523497b4040000, quorum=localhost:55120, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2016-01-11 22:48:24,409 INFO [RpcServer.responder] ipc.RpcServer$Responder(956): RpcServer.responder: starting 2016-01-11 22:48:24,410 INFO [RpcServer.listener,port=49897] ipc.RpcServer$Listener(768): RpcServer.listener,port=49897: starting 2016-01-11 22:48:24,410 DEBUG [main] ipc.RpcExecutor(115): B.default Start Handler index=0 queue=0 2016-01-11 22:48:24,411 DEBUG [main] ipc.RpcExecutor(115): B.default Start Handler index=1 queue=0 2016-01-11 22:48:24,411 DEBUG [main] ipc.RpcExecutor(115): B.default Start Handler index=2 queue=0 2016-01-11 22:48:24,411 DEBUG [main] ipc.RpcExecutor(115): B.default Start Handler index=3 queue=0 2016-01-11 22:48:24,412 DEBUG [main] ipc.RpcExecutor(115): B.default Start Handler index=4 queue=0 2016-01-11 22:48:24,412 DEBUG [main] ipc.RpcExecutor(115): Priority Start Handler index=0 queue=0 2016-01-11 22:48:24,412 DEBUG [main] ipc.RpcExecutor(115): Priority Start Handler index=1 queue=1 2016-01-11 22:48:24,413 DEBUG [main] ipc.RpcExecutor(115): Priority Start Handler index=2 queue=0 2016-01-11 22:48:24,413 DEBUG [main] ipc.RpcExecutor(115): Priority Start Handler index=3 queue=1 2016-01-11 22:48:24,413 DEBUG [main] ipc.RpcExecutor(115): Priority Start Handler index=4 queue=0 2016-01-11 22:48:24,413 DEBUG [main] ipc.RpcExecutor(115): Replication Start Handler index=0 queue=0 2016-01-11 22:48:24,414 DEBUG [main] ipc.RpcExecutor(115): Replication Start Handler index=1 queue=0 2016-01-11 22:48:24,414 DEBUG [main] ipc.RpcExecutor(115): Replication Start Handler index=2 queue=0 2016-01-11 22:48:24,419 INFO [main] master.HMaster(367): hbase.rootdir=hdfs://localhost:59295/user/elliott/test-data/4f0ddc3b-9bf4-4ec6-b2ab-29e91003a033, hbase.cluster.distributed=false 2016-01-11 22:48:24,429 INFO [main] master.HMaster(1705): Adding backup master ZNode /hbase/backup-masters/dev1762.prn2.facebook.com,49897,1452581304150 2016-01-11 22:48:24,452 DEBUG [main] zookeeper.ZKUtil(368): master:49897-0x1523497b4040000, quorum=localhost:55120, baseZNode=/hbase Set watcher on existing znode=/hbase/backup-masters/dev1762.prn2.facebook.com,49897,1452581304150 2016-01-11 22:48:24,459 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(554): master:49897-0x1523497b4040000, quorum=localhost:55120, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2016-01-11 22:48:24,460 DEBUG [dev1762:49897.activeMasterManager] zookeeper.ZKUtil(368): master:49897-0x1523497b4040000, quorum=localhost:55120, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2016-01-11 22:48:24,461 INFO [dev1762:49897.activeMasterManager] master.ActiveMasterManager(170): Deleting ZNode for /hbase/backup-masters/dev1762.prn2.facebook.com,49897,1452581304150 from backup master directory 2016-01-11 22:48:24,467 DEBUG [main-EventThread] zookeeper.ZKUtil(368): master:49897-0x1523497b4040000, quorum=localhost:55120, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2016-01-11 22:48:24,467 DEBUG [main-EventThread] master.ActiveMasterManager(126): A master is now available 2016-01-11 22:48:24,468 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(554): master:49897-0x1523497b4040000, quorum=localhost:55120, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/backup-masters/dev1762.prn2.facebook.com,49897,1452581304150 2016-01-11 22:48:24,468 WARN [dev1762:49897.activeMasterManager] hbase.ZNodeClearer(58): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2016-01-11 22:48:24,468 INFO [dev1762:49897.activeMasterManager] master.ActiveMasterManager(179): Registered Active Master=dev1762.prn2.facebook.com,49897,1452581304150 2016-01-11 22:48:24,508 INFO [main] client.ConnectionUtils(104): regionserver/dev1762.prn2.facebook.com/10.35.11.29:0 server-side HConnection retries=350 2016-01-11 22:48:24,509 INFO [main] ipc.SimpleRpcScheduler(128): Using deadline as user call queue, count=1 2016-01-11 22:48:24,510 INFO [main] ipc.RpcServer$Listener(637): regionserver/dev1762.prn2.facebook.com/10.35.11.29:0: started 3 reader(s) listening on port=61792 2016-01-11 22:48:24,513 INFO [main] fs.HFileSystem(252): Added intercepting call to namenode#getBlockLocations so can do block reordering using class class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2016-01-11 22:48:24,515 INFO [main] zookeeper.RecoverableZooKeeper(120): Process identifier=regionserver:61792 connecting to ZooKeeper ensemble=localhost:55120 2016-01-11 22:48:24,525 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(554): regionserver:617920x0, quorum=localhost:55120, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2016-01-11 22:48:24,527 DEBUG [main] zookeeper.ZKUtil(368): regionserver:617920x0, quorum=localhost:55120, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2016-01-11 22:48:24,528 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(638): regionserver:61792-0x1523497b4040001 connected 2016-01-11 22:48:24,529 DEBUG [main] zookeeper.ZKUtil(370): regionserver:61792-0x1523497b4040001, quorum=localhost:55120, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2016-01-11 22:48:24,529 INFO [RpcServer.responder] ipc.RpcServer$Responder(956): RpcServer.responder: starting 2016-01-11 22:48:24,529 INFO [RpcServer.listener,port=61792] ipc.RpcServer$Listener(768): RpcServer.listener,port=61792: starting 2016-01-11 22:48:24,529 DEBUG [main] ipc.RpcExecutor(115): B.default Start Handler index=0 queue=0 2016-01-11 22:48:24,530 DEBUG [main] ipc.RpcExecutor(115): B.default Start Handler index=1 queue=0 2016-01-11 22:48:24,530 DEBUG [main] ipc.RpcExecutor(115): B.default Start Handler index=2 queue=0 2016-01-11 22:48:24,531 DEBUG [main] ipc.RpcExecutor(115): B.default Start Handler index=3 queue=0 2016-01-11 22:48:24,531 DEBUG [main] ipc.RpcExecutor(115): B.default Start Handler index=4 queue=0 2016-01-11 22:48:24,531 DEBUG [main] ipc.RpcExecutor(115): Priority Start Handler index=0 queue=0 2016-01-11 22:48:24,531 DEBUG [main] ipc.RpcExecutor(115): Priority Start Handler index=1 queue=1 2016-01-11 22:48:24,532 DEBUG [main] ipc.RpcExecutor(115): Priority Start Handler index=2 queue=0 2016-01-11 22:48:24,532 DEBUG [main] ipc.RpcExecutor(115): Priority Start Handler index=3 queue=1 2016-01-11 22:48:24,532 DEBUG [main] ipc.RpcExecutor(115): Priority Start Handler index=4 queue=0 2016-01-11 22:48:24,533 DEBUG [main] ipc.RpcExecutor(115): Replication Start Handler index=0 queue=0 2016-01-11 22:48:24,533 DEBUG [main] ipc.RpcExecutor(115): Replication Start Handler index=1 queue=0 2016-01-11 22:48:24,533 DEBUG [main] ipc.RpcExecutor(115): Replication Start Handler index=2 queue=0 2016-01-11 22:48:24,558 INFO [IPC Server handler 0 on 59295] blockmanagement.BlockManager(2383): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:38785 is added to blk_1073741826_1002{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-4449a44e-3e96-4db5-b41c-f22c79a44f57:NORMAL|RBW], ReplicaUnderConstruction[[DISK]DS-e3b160d5-0860-46e6-8012-66df73514c0f:NORMAL|RBW]]} size 0 2016-01-11 22:48:24,559 INFO [IPC Server handler 2 on 59295] blockmanagement.BlockManager(2383): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:46850 is added to blk_1073741826_1002{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-4449a44e-3e96-4db5-b41c-f22c79a44f57:NORMAL|RBW], ReplicaUnderConstruction[[DISK]DS-e3b160d5-0860-46e6-8012-66df73514c0f:NORMAL|RBW]]} size 0 2016-01-11 22:48:24,560 INFO [main] client.ConnectionUtils(104): regionserver/dev1762.prn2.facebook.com/10.35.11.29:0 server-side HConnection retries=350 2016-01-11 22:48:24,561 INFO [main] ipc.SimpleRpcScheduler(128): Using deadline as user call queue, count=1 2016-01-11 22:48:24,562 INFO [main] ipc.RpcServer$Listener(637): regionserver/dev1762.prn2.facebook.com/10.35.11.29:0: started 3 reader(s) listening on port=33025 2016-01-11 22:48:24,562 DEBUG [dev1762:49897.activeMasterManager] util.FSUtils(899): Created cluster ID file at hdfs://localhost:59295/user/elliott/test-data/4f0ddc3b-9bf4-4ec6-b2ab-29e91003a033/hbase.id with ID: f5b0bbf6-d52e-4b05-bae9-5f7e65466c80 2016-01-11 22:48:24,565 INFO [main] fs.HFileSystem(252): Added intercepting call to namenode#getBlockLocations so can do block reordering using class class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2016-01-11 22:48:24,567 INFO [main] zookeeper.RecoverableZooKeeper(120): Process identifier=regionserver:33025 connecting to ZooKeeper ensemble=localhost:55120 2016-01-11 22:48:24,576 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(554): regionserver:330250x0, quorum=localhost:55120, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2016-01-11 22:48:24,577 DEBUG [main] zookeeper.ZKUtil(368): regionserver:330250x0, quorum=localhost:55120, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2016-01-11 22:48:24,579 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(638): regionserver:33025-0x1523497b4040002 connected 2016-01-11 22:48:24,579 DEBUG [main] zookeeper.ZKUtil(370): regionserver:33025-0x1523497b4040002, quorum=localhost:55120, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2016-01-11 22:48:24,580 INFO [RpcServer.responder] ipc.RpcServer$Responder(956): RpcServer.responder: starting 2016-01-11 22:48:24,580 INFO [RpcServer.listener,port=33025] ipc.RpcServer$Listener(768): RpcServer.listener,port=33025: starting 2016-01-11 22:48:24,580 DEBUG [main] ipc.RpcExecutor(115): B.default Start Handler index=0 queue=0 2016-01-11 22:48:24,581 DEBUG [main] ipc.RpcExecutor(115): B.default Start Handler index=1 queue=0 2016-01-11 22:48:24,581 DEBUG [main] ipc.RpcExecutor(115): B.default Start Handler index=2 queue=0 2016-01-11 22:48:24,581 DEBUG [main] ipc.RpcExecutor(115): B.default Start Handler index=3 queue=0 2016-01-11 22:48:24,581 DEBUG [main] ipc.RpcExecutor(115): B.default Start Handler index=4 queue=0 2016-01-11 22:48:24,582 DEBUG [main] ipc.RpcExecutor(115): Priority Start Handler index=0 queue=0 2016-01-11 22:48:24,582 DEBUG [main] ipc.RpcExecutor(115): Priority Start Handler index=1 queue=1 2016-01-11 22:48:24,582 DEBUG [main] ipc.RpcExecutor(115): Priority Start Handler index=2 queue=0 2016-01-11 22:48:24,583 DEBUG [main] ipc.RpcExecutor(115): Priority Start Handler index=3 queue=1 2016-01-11 22:48:24,583 DEBUG [main] ipc.RpcExecutor(115): Priority Start Handler index=4 queue=0 2016-01-11 22:48:24,583 DEBUG [main] ipc.RpcExecutor(115): Replication Start Handler index=0 queue=0 2016-01-11 22:48:24,583 DEBUG [main] ipc.RpcExecutor(115): Replication Start Handler index=1 queue=0 2016-01-11 22:48:24,584 DEBUG [main] ipc.RpcExecutor(115): Replication Start Handler index=2 queue=0 2016-01-11 22:48:24,600 INFO [dev1762:49897.activeMasterManager] master.MasterFileSystem(531): BOOTSTRAP: creating hbase:meta region 2016-01-11 22:48:24,603 INFO [dev1762:49897.activeMasterManager] regionserver.HRegion(6219): creating HRegion hbase:meta HTD == 'hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}, {NAME => 'info', BLOOMFILTER => 'NONE', VERSIONS => '10', IN_MEMORY => 'false', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', COMPRESSION => 'NONE', MIN_VERSIONS => '0', BLOCKCACHE => 'false', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} RootDir = hdfs://localhost:59295/user/elliott/test-data/4f0ddc3b-9bf4-4ec6-b2ab-29e91003a033 Table name == hbase:meta 2016-01-11 22:48:24,610 INFO [M:0;dev1762:49897] zookeeper.RecoverableZooKeeper(120): Process identifier=hconnection-0x1db27713 connecting to ZooKeeper ensemble=localhost:55120 2016-01-11 22:48:24,610 INFO [RS:0;dev1762:61792] zookeeper.RecoverableZooKeeper(120): Process identifier=hconnection-0x3e7515aa connecting to ZooKeeper ensemble=localhost:55120 2016-01-11 22:48:24,610 INFO [RS:1;dev1762:33025] zookeeper.RecoverableZooKeeper(120): Process identifier=hconnection-0x4dd24369 connecting to ZooKeeper ensemble=localhost:55120 2016-01-11 22:48:24,617 DEBUG [M:0;dev1762:49897-EventThread] zookeeper.ZooKeeperWatcher(554): hconnection-0x1db277130x0, quorum=localhost:55120, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2016-01-11 22:48:24,619 DEBUG [M:0;dev1762:49897-EventThread] zookeeper.ZooKeeperWatcher(638): hconnection-0x1db27713-0x1523497b4040003 connected 2016-01-11 22:48:24,626 DEBUG [RS:1;dev1762:33025-EventThread] zookeeper.ZooKeeperWatcher(554): hconnection-0x4dd243690x0, quorum=localhost:55120, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2016-01-11 22:48:24,627 DEBUG [RS:0;dev1762:61792-EventThread] zookeeper.ZooKeeperWatcher(554): hconnection-0x3e7515aa0x0, quorum=localhost:55120, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2016-01-11 22:48:24,628 DEBUG [RS:1;dev1762:33025-EventThread] zookeeper.ZooKeeperWatcher(638): hconnection-0x4dd24369-0x1523497b4040004 connected 2016-01-11 22:48:24,628 INFO [RS:0;dev1762:61792] client.ZooKeeperRegistry(107): ClusterId read in ZooKeeper is null 2016-01-11 22:48:24,628 INFO [M:0;dev1762:49897] client.ZooKeeperRegistry(107): ClusterId read in ZooKeeper is null 2016-01-11 22:48:24,629 DEBUG [RS:0;dev1762:61792] client.ConnectionManager$HConnectionImplementation(911): clusterid came back null, using default default-cluster 2016-01-11 22:48:24,628 INFO [RS:1;dev1762:33025] client.ZooKeeperRegistry(107): ClusterId read in ZooKeeper is null 2016-01-11 22:48:24,630 DEBUG [M:0;dev1762:49897] client.ConnectionManager$HConnectionImplementation(911): clusterid came back null, using default default-cluster 2016-01-11 22:48:24,630 DEBUG [RS:0;dev1762:61792-EventThread] zookeeper.ZooKeeperWatcher(638): hconnection-0x3e7515aa-0x1523497b4040005 connected 2016-01-11 22:48:24,630 DEBUG [RS:1;dev1762:33025] client.ConnectionManager$HConnectionImplementation(911): clusterid came back null, using default default-cluster 2016-01-11 22:48:24,632 INFO [IPC Server handler 4 on 59295] blockmanagement.BlockManager(2383): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:38785 is added to blk_1073741827_1003{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-2f4161bb-a789-4c3e-bb22-3dbf1babdf25:NORMAL|RBW], ReplicaUnderConstruction[[DISK]DS-940d5c3a-1db1-4402-9e4c-a19c2cc6af02:NORMAL|RBW]]} size 0 2016-01-11 22:48:24,633 INFO [IPC Server handler 9 on 59295] blockmanagement.BlockManager(2383): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:46850 is added to blk_1073741827_1003 size 32 2016-01-11 22:48:24,636 DEBUG [dev1762:49897.activeMasterManager] regionserver.HRegion(733): Instantiated hbase:meta,,1.1588230740 2016-01-11 22:48:24,638 DEBUG [RS:0;dev1762:61792] ipc.AbstractRpcClient(115): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@67c165f1, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2016-01-11 22:48:24,638 DEBUG [RS:1;dev1762:33025] ipc.AbstractRpcClient(115): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4315b4a3, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2016-01-11 22:48:24,638 DEBUG [M:0;dev1762:49897] ipc.AbstractRpcClient(115): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@13759ad5, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2016-01-11 22:48:24,677 INFO [StoreOpener-1588230740-1] hfile.CacheConfig(548): Allocating LruBlockCache size=995.60 MB, blockSize=64 KB 2016-01-11 22:48:24,687 DEBUG [StoreOpener-1588230740-1] hfile.CacheConfig(562): Trying to use Internal l2 cache 2016-01-11 22:48:24,687 INFO [StoreOpener-1588230740-1] hfile.CacheConfig(292): blockCache=LruBlockCache{blockCount=0, currentSize=1071840, freeSize=1042890464, maxSize=1043962304, heapSize=1071840, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=false, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2016-01-11 22:48:24,694 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(104): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000 2016-01-11 22:48:24,698 DEBUG [StoreOpener-1588230740-1] regionserver.HRegionFileSystem(201): No StoreFiles for: hdfs://localhost:59295/user/elliott/test-data/4f0ddc3b-9bf4-4ec6-b2ab-29e91003a033/data/hbase/meta/1588230740/info 2016-01-11 22:48:24,711 DEBUG [dev1762:49897.activeMasterManager] regionserver.HRegion(3890): Found 0 recovered edits file(s) under hdfs://localhost:59295/user/elliott/test-data/4f0ddc3b-9bf4-4ec6-b2ab-29e91003a033/data/hbase/meta/1588230740 2016-01-11 22:48:24,722 DEBUG [dev1762:49897.activeMasterManager] regionserver.FlushLargeStoresPolicy(56): hbase.hregion.percolumnfamilyflush.size.lower.bound is not specified, use global config(16777216) instead 2016-01-11 22:48:24,727 DEBUG [dev1762:49897.activeMasterManager] wal.WALSplitter(729): Wrote region seqId=hdfs://localhost:59295/user/elliott/test-data/4f0ddc3b-9bf4-4ec6-b2ab-29e91003a033/data/hbase/meta/1588230740/recovered.edits/2.seqid to file, newSeqId=2, maxSeqId=0 2016-01-11 22:48:24,728 INFO [dev1762:49897.activeMasterManager] regionserver.HRegion(868): Onlined 1588230740; next sequenceid=2 2016-01-11 22:48:24,728 DEBUG [dev1762:49897.activeMasterManager] regionserver.HRegion(1389): Closing hbase:meta,,1.1588230740: disabling compactions & flushes 2016-01-11 22:48:24,728 DEBUG [dev1762:49897.activeMasterManager] regionserver.HRegion(1416): Updates disabled for region hbase:meta,,1.1588230740 2016-01-11 22:48:24,729 INFO [StoreCloserThread-hbase:meta,,1.1588230740-1] regionserver.HStore(875): Closed info 2016-01-11 22:48:24,729 INFO [dev1762:49897.activeMasterManager] regionserver.HRegion(1519): Closed hbase:meta,,1.1588230740 2016-01-11 22:48:24,752 INFO [IPC Server handler 8 on 59295] blockmanagement.BlockManager(2383): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:46850 is added to blk_1073741828_1004{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-e3b160d5-0860-46e6-8012-66df73514c0f:NORMAL|RBW], ReplicaUnderConstruction[[DISK]DS-4449a44e-3e96-4db5-b41c-f22c79a44f57:NORMAL|RBW]]} size 0 2016-01-11 22:48:24,753 INFO [IPC Server handler 9 on 59295] blockmanagement.BlockManager(2383): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:38785 is added to blk_1073741828_1004{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-e3b160d5-0860-46e6-8012-66df73514c0f:NORMAL|RBW], ReplicaUnderConstruction[[DISK]DS-4449a44e-3e96-4db5-b41c-f22c79a44f57:NORMAL|RBW]]} size 0 2016-01-11 22:48:24,756 DEBUG [dev1762:49897.activeMasterManager] util.FSTableDescriptors(656): Wrote descriptor into: hdfs://localhost:59295/user/elliott/test-data/4f0ddc3b-9bf4-4ec6-b2ab-29e91003a033/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2016-01-11 22:48:24,769 INFO [dev1762:49897.activeMasterManager] fs.HFileSystem(252): Added intercepting call to namenode#getBlockLocations so can do block reordering using class class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2016-01-11 22:48:24,774 INFO [dev1762:49897.activeMasterManager] coordination.ZKSplitLogManagerCoordination(599): Found 0 orphan tasks and 0 rescan nodes 2016-01-11 22:48:24,775 DEBUG [dev1762:49897.activeMasterManager] util.FSTableDescriptors(204): Fetching table descriptors from the filesystem. 2016-01-11 22:48:24,789 INFO [dev1762:49897.activeMasterManager] zookeeper.RecoverableZooKeeper(120): Process identifier=hconnection-0x194f7d4b connecting to ZooKeeper ensemble=localhost:55120 2016-01-11 22:48:24,800 DEBUG [dev1762:49897.activeMasterManager-EventThread] zookeeper.ZooKeeperWatcher(554): hconnection-0x194f7d4b0x0, quorum=localhost:55120, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2016-01-11 22:48:24,801 DEBUG [dev1762:49897.activeMasterManager] ipc.AbstractRpcClient(115): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@15d54f0b, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2016-01-11 22:48:24,802 DEBUG [dev1762:49897.activeMasterManager-EventThread] zookeeper.ZooKeeperWatcher(638): hconnection-0x194f7d4b-0x1523497b4040006 connected 2016-01-11 22:48:24,810 INFO [dev1762:49897.activeMasterManager] balancer.StochasticLoadBalancer(135): loading config 2016-01-11 22:48:24,817 DEBUG [dev1762:49897.activeMasterManager] zookeeper.ZKUtil(370): master:49897-0x1523497b4040000, quorum=localhost:55120, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/balancer 2016-01-11 22:48:24,818 DEBUG [dev1762:49897.activeMasterManager] zookeeper.ZKUtil(370): master:49897-0x1523497b4040000, quorum=localhost:55120, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/normalizer 2016-01-11 22:48:24,850 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(554): master:49897-0x1523497b4040000, quorum=localhost:55120, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2016-01-11 22:48:24,850 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(554): regionserver:33025-0x1523497b4040002, quorum=localhost:55120, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2016-01-11 22:48:24,850 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(554): regionserver:61792-0x1523497b4040001, quorum=localhost:55120, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2016-01-11 22:48:24,851 INFO [dev1762:49897.activeMasterManager] master.HMaster(586): Server active/primary master=dev1762.prn2.facebook.com,49897,1452581304150, sessionid=0x1523497b4040000, setting cluster-up flag (Was=false) 2016-01-11 22:48:24,853 INFO [M:0;dev1762:49897] regionserver.HRegionServer(782): ClusterId : f5b0bbf6-d52e-4b05-bae9-5f7e65466c80 2016-01-11 22:48:24,853 INFO [RS:1;dev1762:33025] regionserver.HRegionServer(782): ClusterId : f5b0bbf6-d52e-4b05-bae9-5f7e65466c80 2016-01-11 22:48:24,854 INFO [RS:0;dev1762:61792] regionserver.HRegionServer(782): ClusterId : f5b0bbf6-d52e-4b05-bae9-5f7e65466c80 2016-01-11 22:48:24,857 DEBUG [RS:0;dev1762:61792] procedure.RegionServerProcedureManagerHost(44): Procedure flush-table-proc is initializing 2016-01-11 22:48:24,857 DEBUG [RS:1;dev1762:33025] procedure.RegionServerProcedureManagerHost(44): Procedure flush-table-proc is initializing 2016-01-11 22:48:24,884 DEBUG [RS:1;dev1762:33025] zookeeper.RecoverableZooKeeper(594): Node /hbase/flush-table-proc already exists 2016-01-11 22:48:24,892 DEBUG [dev1762:49897.activeMasterManager] zookeeper.RecoverableZooKeeper(594): Node /hbase/flush-table-proc already exists 2016-01-11 22:48:24,901 DEBUG [RS:1;dev1762:33025] zookeeper.RecoverableZooKeeper(594): Node /hbase/flush-table-proc/acquired already exists 2016-01-11 22:48:24,908 DEBUG [dev1762:49897.activeMasterManager] zookeeper.RecoverableZooKeeper(594): Node /hbase/flush-table-proc/acquired already exists 2016-01-11 22:48:24,925 DEBUG [RS:1;dev1762:33025] zookeeper.RecoverableZooKeeper(594): Node /hbase/flush-table-proc/reached already exists 2016-01-11 22:48:24,935 DEBUG [RS:0;dev1762:61792] procedure.RegionServerProcedureManagerHost(46): Procedure flush-table-proc is initialized 2016-01-11 22:48:24,935 DEBUG [RS:1;dev1762:33025] procedure.RegionServerProcedureManagerHost(46): Procedure flush-table-proc is initialized 2016-01-11 22:48:24,935 DEBUG [RS:1;dev1762:33025] procedure.RegionServerProcedureManagerHost(44): Procedure online-snapshot is initializing 2016-01-11 22:48:24,935 DEBUG [RS:0;dev1762:61792] procedure.RegionServerProcedureManagerHost(44): Procedure online-snapshot is initializing 2016-01-11 22:48:24,942 DEBUG [dev1762:49897.activeMasterManager] zookeeper.RecoverableZooKeeper(594): Node /hbase/flush-table-proc/abort already exists 2016-01-11 22:48:24,942 INFO [dev1762:49897.activeMasterManager] procedure.ZKProcedureUtil(270): Clearing all procedure znodes: /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2016-01-11 22:48:24,967 DEBUG [RS:0;dev1762:61792] zookeeper.RecoverableZooKeeper(594): Node /hbase/online-snapshot already exists 2016-01-11 22:48:24,976 DEBUG [dev1762:49897.activeMasterManager] procedure.ZKProcedureCoordinatorRpcs(248): Starting the controller for procedure member:dev1762.prn2.facebook.com,49897,1452581304150 2016-01-11 22:48:24,983 DEBUG [RS:0;dev1762:61792] zookeeper.RecoverableZooKeeper(594): Node /hbase/online-snapshot/acquired already exists 2016-01-11 22:48:24,992 DEBUG [dev1762:49897.activeMasterManager] zookeeper.RecoverableZooKeeper(594): Node /hbase/online-snapshot/acquired already exists 2016-01-11 22:48:25,009 DEBUG [RS:1;dev1762:33025] procedure.RegionServerProcedureManagerHost(46): Procedure online-snapshot is initialized 2016-01-11 22:48:25,013 INFO [RS:1;dev1762:33025] regionserver.MemStoreFlusher(125): globalMemStoreLimit=995.6 M, globalMemStoreLimitLowMark=945.8 M, maxHeap=2.4 G 2016-01-11 22:48:25,017 DEBUG [RS:0;dev1762:61792] zookeeper.RecoverableZooKeeper(594): Node /hbase/online-snapshot/abort already exists 2016-01-11 22:48:25,017 DEBUG [dev1762:49897.activeMasterManager] zookeeper.RecoverableZooKeeper(594): Node /hbase/online-snapshot/abort already exists 2016-01-11 22:48:25,017 DEBUG [RS:0;dev1762:61792] procedure.RegionServerProcedureManagerHost(46): Procedure online-snapshot is initialized 2016-01-11 22:48:25,017 INFO [dev1762:49897.activeMasterManager] procedure.ZKProcedureUtil(270): Clearing all procedure znodes: /hbase/online-snapshot/acquired /hbase/online-snapshot/reached /hbase/online-snapshot/abort 2016-01-11 22:48:25,018 INFO [RS:0;dev1762:61792] regionserver.MemStoreFlusher(125): globalMemStoreLimit=995.6 M, globalMemStoreLimitLowMark=945.8 M, maxHeap=2.4 G 2016-01-11 22:48:25,019 INFO [RS:1;dev1762:33025] regionserver.HRegionServer$CompactionChecker(1523): CompactionChecker runs every 1sec 2016-01-11 22:48:25,019 INFO [RS:0;dev1762:61792] regionserver.HRegionServer$CompactionChecker(1523): CompactionChecker runs every 1sec 2016-01-11 22:48:25,019 DEBUG [dev1762:49897.activeMasterManager] procedure.ZKProcedureCoordinatorRpcs(248): Starting the controller for procedure member:dev1762.prn2.facebook.com,49897,1452581304150 2016-01-11 22:48:25,022 DEBUG [RS:0;dev1762:61792] ipc.AbstractRpcClient(115): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1f902a23, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=dev1762.prn2.facebook.com/10.35.11.29:0 2016-01-11 22:48:25,022 DEBUG [RS:1;dev1762:33025] ipc.AbstractRpcClient(115): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@33e07b17, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=dev1762.prn2.facebook.com/10.35.11.29:0 2016-01-11 22:48:25,025 DEBUG [RS:0;dev1762:61792] regionserver.ShutdownHook(87): Installed shutdown hook thread: Shutdownhook:RS:0;dev1762:61792 2016-01-11 22:48:25,025 DEBUG [RS:1;dev1762:33025] regionserver.ShutdownHook(87): Installed shutdown hook thread: Shutdownhook:RS:1;dev1762:33025 2016-01-11 22:48:25,045 INFO [dev1762:49897.activeMasterManager] master.MasterCoprocessorHost(86): System coprocessor loading is enabled 2016-01-11 22:48:25,046 DEBUG [dev1762:49897.activeMasterManager] executor.ExecutorService(99): Starting executor service name=MASTER_OPEN_REGION-dev1762:49897, corePoolSize=5, maxPoolSize=5 2016-01-11 22:48:25,046 DEBUG [dev1762:49897.activeMasterManager] executor.ExecutorService(99): Starting executor service name=MASTER_CLOSE_REGION-dev1762:49897, corePoolSize=5, maxPoolSize=5 2016-01-11 22:48:25,046 DEBUG [dev1762:49897.activeMasterManager] executor.ExecutorService(99): Starting executor service name=MASTER_SERVER_OPERATIONS-dev1762:49897, corePoolSize=5, maxPoolSize=5 2016-01-11 22:48:25,046 DEBUG [dev1762:49897.activeMasterManager] executor.ExecutorService(99): Starting executor service name=MASTER_META_SERVER_OPERATIONS-dev1762:49897, corePoolSize=5, maxPoolSize=5 2016-01-11 22:48:25,046 DEBUG [dev1762:49897.activeMasterManager] executor.ExecutorService(99): Starting executor service name=M_LOG_REPLAY_OPS-dev1762:49897, corePoolSize=10, maxPoolSize=10 2016-01-11 22:48:25,047 DEBUG [dev1762:49897.activeMasterManager] executor.ExecutorService(99): Starting executor service name=MASTER_TABLE_OPERATIONS-dev1762:49897, corePoolSize=1, maxPoolSize=1 2016-01-11 22:48:25,058 INFO [dev1762:49897.activeMasterManager] procedure2.ProcedureExecutor(472): Starting procedure executor threads=33 2016-01-11 22:48:25,059 INFO [dev1762:49897.activeMasterManager] wal.WALProcedureStore(251): Starting WAL Procedure Store lease recovery 2016-01-11 22:48:25,060 WARN [dev1762:49897.activeMasterManager] wal.WALProcedureStore(879): Log directory not found: File hdfs://localhost:59295/user/elliott/test-data/4f0ddc3b-9bf4-4ec6-b2ab-29e91003a033/MasterProcWALs does not exist. 2016-01-11 22:48:25,066 DEBUG [dev1762:49897.activeMasterManager] wal.WALProcedureStore(772): Roll new state log: 1 2016-01-11 22:48:25,067 INFO [dev1762:49897.activeMasterManager] wal.WALProcedureStore(274): Lease acquired for flushLogId: 1 2016-01-11 22:48:25,068 DEBUG [dev1762:49897.activeMasterManager] wal.WALProcedureStore(291): No state logs to replay. 2016-01-11 22:48:25,068 DEBUG [dev1762:49897.activeMasterManager] procedure2.ProcedureExecutor$1(298): load procedures maxProcId=0 2016-01-11 22:48:25,071 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(554): master:49897-0x1523497b4040000, quorum=localhost:55120, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2016-01-11 22:48:25,075 DEBUG [dev1762:49897.activeMasterManager] cleaner.CleanerChore(91): initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveLogCleaner 2016-01-11 22:48:25,076 INFO [dev1762:49897.activeMasterManager] zookeeper.RecoverableZooKeeper(120): Process identifier=replicationLogCleaner connecting to ZooKeeper ensemble=localhost:55120 2016-01-11 22:48:25,084 DEBUG [RS:1;dev1762:33025] zookeeper.ZKUtil(368): regionserver:33025-0x1523497b4040002, quorum=localhost:55120, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/dev1762.prn2.facebook.com,33025,1452581304563 2016-01-11 22:48:25,087 INFO [RS:1;dev1762:33025] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2016-01-11 22:48:25,087 INFO [RS:1;dev1762:33025] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2016-01-11 22:48:25,088 INFO [RS:1;dev1762:33025] regionserver.HRegionServer(2276): reportForDuty to master=dev1762.prn2.facebook.com,49897,1452581304150 with port=33025, startcode=1452581304563 2016-01-11 22:48:25,092 DEBUG [dev1762:49897.activeMasterManager-EventThread] zookeeper.ZooKeeperWatcher(554): replicationLogCleaner0x0, quorum=localhost:55120, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2016-01-11 22:48:25,092 DEBUG [RS:0;dev1762:61792] zookeeper.ZKUtil(368): regionserver:61792-0x1523497b4040001, quorum=localhost:55120, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/dev1762.prn2.facebook.com,61792,1452581304511 2016-01-11 22:48:25,092 DEBUG [main-EventThread] zookeeper.ZKUtil(368): master:49897-0x1523497b4040000, quorum=localhost:55120, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/dev1762.prn2.facebook.com,61792,1452581304511 2016-01-11 22:48:25,093 INFO [RS:0;dev1762:61792] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2016-01-11 22:48:25,093 INFO [RS:0;dev1762:61792] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2016-01-11 22:48:25,093 INFO [RS:0;dev1762:61792] regionserver.HRegionServer(2276): reportForDuty to master=dev1762.prn2.facebook.com,49897,1452581304150 with port=61792, startcode=1452581304511 2016-01-11 22:48:25,094 DEBUG [dev1762:49897.activeMasterManager-EventThread] zookeeper.ZooKeeperWatcher(638): replicationLogCleaner-0x1523497b4040007 connected 2016-01-11 22:48:25,100 DEBUG [main-EventThread] zookeeper.ZKUtil(368): master:49897-0x1523497b4040000, quorum=localhost:55120, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/dev1762.prn2.facebook.com,33025,1452581304563 2016-01-11 22:48:25,109 DEBUG [main-EventThread] zookeeper.RegionServerTracker(93): Added tracking of RS /hbase/rs/dev1762.prn2.facebook.com,61792,1452581304511 2016-01-11 22:48:25,117 DEBUG [dev1762:49897.activeMasterManager] cleaner.CleanerChore(91): initialize cleaner=org.apache.hadoop.hbase.replication.master.ReplicationLogCleaner 2016-01-11 22:48:25,117 DEBUG [main-EventThread] zookeeper.RegionServerTracker(93): Added tracking of RS /hbase/rs/dev1762.prn2.facebook.com,33025,1452581304563 2016-01-11 22:48:25,119 DEBUG [dev1762:49897.activeMasterManager] cleaner.CleanerChore(91): initialize cleaner=org.apache.hadoop.hbase.master.snapshot.SnapshotLogCleaner 2016-01-11 22:48:25,122 DEBUG [dev1762:49897.activeMasterManager] cleaner.CleanerChore(91): initialize cleaner=org.apache.hadoop.hbase.master.cleaner.HFileLinkCleaner 2016-01-11 22:48:25,123 DEBUG [dev1762:49897.activeMasterManager] cleaner.CleanerChore(91): initialize cleaner=org.apache.hadoop.hbase.master.snapshot.SnapshotHFileCleaner 2016-01-11 22:48:25,124 DEBUG [dev1762:49897.activeMasterManager] cleaner.CleanerChore(91): initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveHFileCleaner 2016-01-11 22:48:25,124 INFO [dev1762:49897.activeMasterManager] master.ServerManager(1028): Waiting for region servers count to settle; currently checked in 0, slept for 0 ms, expecting minimum of 2, maximum of 2, timeout of 4500 ms, interval of 1500 ms. 2016-01-11 22:48:25,131 DEBUG [RS:1;dev1762:33025] ipc.RpcClientImpl$Connection(351): Use SIMPLE authentication for service RegionServerStatusService, sasl=false 2016-01-11 22:48:25,131 DEBUG [RS:0;dev1762:61792] ipc.RpcClientImpl$Connection(351): Use SIMPLE authentication for service RegionServerStatusService, sasl=false 2016-01-11 22:48:25,143 DEBUG [RS:0;dev1762:61792] ipc.RpcClientImpl$Connection(716): Connecting to dev1762.prn2.facebook.com/10.35.11.29:49897 2016-01-11 22:48:25,143 DEBUG [RS:1;dev1762:33025] ipc.RpcClientImpl$Connection(716): Connecting to dev1762.prn2.facebook.com/10.35.11.29:49897 2016-01-11 22:48:25,150 DEBUG [RpcServer.listener,port=49897] ipc.RpcServer$Listener(879): RpcServer.listener,port=49897: connection from 10.35.11.29:45302; # active connections: 1 2016-01-11 22:48:25,151 DEBUG [RpcServer.listener,port=49897] ipc.RpcServer$Listener(879): RpcServer.listener,port=49897: connection from 10.35.11.29:56116; # active connections: 2 2016-01-11 22:48:25,152 INFO [RpcServer.reader=1,bindAddress=dev1762.prn2.facebook.com,port=49897] ipc.RpcServer$Connection(1712): Auth successful for elliott.hfs.0 (auth:SIMPLE) 2016-01-11 22:48:25,152 INFO [RpcServer.reader=2,bindAddress=dev1762.prn2.facebook.com,port=49897] ipc.RpcServer$Connection(1712): Auth successful for elliott.hfs.1 (auth:SIMPLE) 2016-01-11 22:48:25,157 INFO [RpcServer.reader=1,bindAddress=dev1762.prn2.facebook.com,port=49897] ipc.RpcServer$Connection(1742): Connection from 10.35.11.29 port: 45302 with version info: version: "1.2.0" url: "git://dev1762.prn2.facebook.com/data/users/elliott/hbase" revision: "4b59f62c231a39fd4a824c0cbf4c2166c4ec2a76" user: "elliott" date: "Mon Jan 11 22:47:43 PST 2016" src_checksum: "58511d3ee667dd7df285c172560770ca" 2016-01-11 22:48:25,157 INFO [RpcServer.reader=2,bindAddress=dev1762.prn2.facebook.com,port=49897] ipc.RpcServer$Connection(1742): Connection from 10.35.11.29 port: 56116 with version info: version: "1.2.0" url: "git://dev1762.prn2.facebook.com/data/users/elliott/hbase" revision: "4b59f62c231a39fd4a824c0cbf4c2166c4ec2a76" user: "elliott" date: "Mon Jan 11 22:47:43 PST 2016" src_checksum: "58511d3ee667dd7df285c172560770ca" 2016-01-11 22:48:25,168 INFO [B.defaultRpcServer.handler=1,queue=0,port=49897] master.ServerManager(459): Registering server=dev1762.prn2.facebook.com,61792,1452581304511 2016-01-11 22:48:25,169 INFO [B.defaultRpcServer.handler=0,queue=0,port=49897] master.ServerManager(459): Registering server=dev1762.prn2.facebook.com,33025,1452581304563 2016-01-11 22:48:25,174 INFO [dev1762:49897.activeMasterManager] master.ServerManager(1045): Finished waiting for region servers count to settle; checked in 2, slept for 50 ms, expecting minimum of 2, maximum of 2, master is running 2016-01-11 22:48:25,177 DEBUG [dev1762:49897.activeMasterManager] zookeeper.ZKUtil(627): master:49897-0x1523497b4040000, quorum=localhost:55120, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-01-11 22:48:25,179 INFO [RS:0;dev1762:61792] regionserver.HRegionServer(1359): Config from master: hbase.rootdir=hdfs://localhost:59295/user/elliott/test-data/4f0ddc3b-9bf4-4ec6-b2ab-29e91003a033 2016-01-11 22:48:25,179 INFO [RS:1;dev1762:33025] regionserver.HRegionServer(1359): Config from master: hbase.rootdir=hdfs://localhost:59295/user/elliott/test-data/4f0ddc3b-9bf4-4ec6-b2ab-29e91003a033 2016-01-11 22:48:25,179 INFO [RS:0;dev1762:61792] regionserver.HRegionServer(1359): Config from master: fs.defaultFS=hdfs://localhost:59295 2016-01-11 22:48:25,180 INFO [RS:1;dev1762:33025] regionserver.HRegionServer(1359): Config from master: fs.defaultFS=hdfs://localhost:59295 2016-01-11 22:48:25,180 INFO [RS:0;dev1762:61792] regionserver.HRegionServer(1359): Config from master: hbase.master.info.port=-1 2016-01-11 22:48:25,180 INFO [RS:1;dev1762:33025] regionserver.HRegionServer(1359): Config from master: hbase.master.info.port=-1 2016-01-11 22:48:25,180 WARN [RS:0;dev1762:61792] hbase.ZNodeClearer(58): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2016-01-11 22:48:25,180 WARN [RS:1;dev1762:33025] hbase.ZNodeClearer(58): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2016-01-11 22:48:25,180 INFO [RS:0;dev1762:61792] hfile.CacheConfig(292): blockCache=LruBlockCache{blockCount=0, currentSize=1071840, freeSize=1042890464, maxSize=1043962304, heapSize=1071840, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2016-01-11 22:48:25,181 INFO [RS:1;dev1762:33025] hfile.CacheConfig(292): blockCache=LruBlockCache{blockCount=0, currentSize=1071840, freeSize=1042890464, maxSize=1043962304, heapSize=1071840, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2016-01-11 22:48:25,181 DEBUG [RS:0;dev1762:61792] regionserver.HRegionServer(1622): logdir=hdfs://localhost:59295/user/elliott/test-data/4f0ddc3b-9bf4-4ec6-b2ab-29e91003a033/WALs/dev1762.prn2.facebook.com,61792,1452581304511 2016-01-11 22:48:25,181 DEBUG [RS:1;dev1762:33025] regionserver.HRegionServer(1622): logdir=hdfs://localhost:59295/user/elliott/test-data/4f0ddc3b-9bf4-4ec6-b2ab-29e91003a033/WALs/dev1762.prn2.facebook.com,33025,1452581304563 2016-01-11 22:48:25,183 DEBUG [dev1762:49897.activeMasterManager] zookeeper.ZKUtil(627): master:49897-0x1523497b4040000, quorum=localhost:55120, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-01-11 22:48:25,192 DEBUG [dev1762:49897.activeMasterManager] zookeeper.ZKUtil(627): master:49897-0x1523497b4040000, quorum=localhost:55120, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-01-11 22:48:25,208 DEBUG [RS:1;dev1762:33025] zookeeper.RecoverableZooKeeper(594): Node /hbase/replication/peers already exists 2016-01-11 22:48:25,213 DEBUG [RS:1;dev1762:33025] regionserver.Replication(143): ReplicationStatisticsThread 300 2016-01-11 22:48:25,213 DEBUG [RS:0;dev1762:61792] regionserver.Replication(143): ReplicationStatisticsThread 300 2016-01-11 22:48:25,222 INFO [RS:0;dev1762:61792] wal.WALFactory(144): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.DefaultWALProvider 2016-01-11 22:48:25,222 INFO [RS:1;dev1762:33025] wal.WALFactory(144): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.DefaultWALProvider 2016-01-11 22:48:25,237 INFO [RS:0;dev1762:61792] wal.FSHLog(529): WAL configuration: blocksize=128 MB, rollsize=121.60 MB, prefix=dev1762.prn2.facebook.com%2C61792%2C1452581304511.default, suffix=, logDir=hdfs://localhost:59295/user/elliott/test-data/4f0ddc3b-9bf4-4ec6-b2ab-29e91003a033/WALs/dev1762.prn2.facebook.com,61792,1452581304511, archiveDir=hdfs://localhost:59295/user/elliott/test-data/4f0ddc3b-9bf4-4ec6-b2ab-29e91003a033/oldWALs 2016-01-11 22:48:25,237 INFO [RS:1;dev1762:33025] wal.FSHLog(529): WAL configuration: blocksize=128 MB, rollsize=121.60 MB, prefix=dev1762.prn2.facebook.com%2C33025%2C1452581304563.default, suffix=, logDir=hdfs://localhost:59295/user/elliott/test-data/4f0ddc3b-9bf4-4ec6-b2ab-29e91003a033/WALs/dev1762.prn2.facebook.com,33025,1452581304563, archiveDir=hdfs://localhost:59295/user/elliott/test-data/4f0ddc3b-9bf4-4ec6-b2ab-29e91003a033/oldWALs 2016-01-11 22:48:25,266 INFO [RS:1;dev1762:33025] wal.FSHLog(1397): Slow sync cost: 17 ms, current pipeline: [] 2016-01-11 22:48:25,266 INFO [RS:0;dev1762:61792] wal.FSHLog(1397): Slow sync cost: 17 ms, current pipeline: [] 2016-01-11 22:48:25,267 INFO [RS:0;dev1762:61792] wal.FSHLog(876): New WAL /user/elliott/test-data/4f0ddc3b-9bf4-4ec6-b2ab-29e91003a033/WALs/dev1762.prn2.facebook.com,61792,1452581304511/dev1762.prn2.facebook.com%2C61792%2C1452581304511.default.1452581305237 2016-01-11 22:48:25,267 INFO [RS:1;dev1762:33025] wal.FSHLog(876): New WAL /user/elliott/test-data/4f0ddc3b-9bf4-4ec6-b2ab-29e91003a033/WALs/dev1762.prn2.facebook.com,33025,1452581304563/dev1762.prn2.facebook.com%2C33025%2C1452581304563.default.1452581305237 2016-01-11 22:48:25,294 INFO [RS:0;dev1762:61792] regionserver.MetricsRegionServerWrapperImpl(101): Computing regionserver metrics every 5000 milliseconds 2016-01-11 22:48:25,294 INFO [RS:1;dev1762:33025] regionserver.MetricsRegionServerWrapperImpl(101): Computing regionserver metrics every 5000 milliseconds 2016-01-11 22:48:25,301 DEBUG [RS:0;dev1762:61792] executor.ExecutorService(99): Starting executor service name=RS_OPEN_REGION-dev1762:61792, corePoolSize=3, maxPoolSize=3 2016-01-11 22:48:25,307 DEBUG [RS:0;dev1762:61792] executor.ExecutorService(99): Starting executor service name=RS_OPEN_META-dev1762:61792, corePoolSize=1, maxPoolSize=1 2016-01-11 22:48:25,307 DEBUG [RS:1;dev1762:33025] executor.ExecutorService(99): Starting executor service name=RS_OPEN_REGION-dev1762:33025, corePoolSize=3, maxPoolSize=3 2016-01-11 22:48:25,308 DEBUG [RS:0;dev1762:61792] executor.ExecutorService(99): Starting executor service name=RS_CLOSE_REGION-dev1762:61792, corePoolSize=3, maxPoolSize=3 2016-01-11 22:48:25,308 DEBUG [RS:1;dev1762:33025] executor.ExecutorService(99): Starting executor service name=RS_OPEN_META-dev1762:33025, corePoolSize=1, maxPoolSize=1 2016-01-11 22:48:25,308 DEBUG [RS:0;dev1762:61792] executor.ExecutorService(99): Starting executor service name=RS_CLOSE_META-dev1762:61792, corePoolSize=1, maxPoolSize=1 2016-01-11 22:48:25,308 DEBUG [RS:1;dev1762:33025] executor.ExecutorService(99): Starting executor service name=RS_CLOSE_REGION-dev1762:33025, corePoolSize=3, maxPoolSize=3 2016-01-11 22:48:25,308 DEBUG [RS:0;dev1762:61792] executor.ExecutorService(99): Starting executor service name=RS_LOG_REPLAY_OPS-dev1762:61792, corePoolSize=2, maxPoolSize=2 2016-01-11 22:48:25,308 DEBUG [RS:1;dev1762:33025] executor.ExecutorService(99): Starting executor service name=RS_CLOSE_META-dev1762:33025, corePoolSize=1, maxPoolSize=1 2016-01-11 22:48:25,308 DEBUG [RS:0;dev1762:61792] executor.ExecutorService(99): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-dev1762:61792, corePoolSize=3, maxPoolSize=3 2016-01-11 22:48:25,308 DEBUG [RS:1;dev1762:33025] executor.ExecutorService(99): Starting executor service name=RS_LOG_REPLAY_OPS-dev1762:33025, corePoolSize=2, maxPoolSize=2 2016-01-11 22:48:25,308 DEBUG [RS:1;dev1762:33025] executor.ExecutorService(99): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-dev1762:33025, corePoolSize=3, maxPoolSize=3 2016-01-11 22:48:25,311 DEBUG [RS:0;dev1762:61792] zookeeper.ZKUtil(368): regionserver:61792-0x1523497b4040001, quorum=localhost:55120, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/dev1762.prn2.facebook.com,61792,1452581304511 2016-01-11 22:48:25,311 DEBUG [RS:1;dev1762:33025] zookeeper.ZKUtil(368): regionserver:33025-0x1523497b4040002, quorum=localhost:55120, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/dev1762.prn2.facebook.com,61792,1452581304511 2016-01-11 22:48:25,311 DEBUG [RS:0;dev1762:61792] zookeeper.ZKUtil(368): regionserver:61792-0x1523497b4040001, quorum=localhost:55120, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/dev1762.prn2.facebook.com,33025,1452581304563 2016-01-11 22:48:25,312 DEBUG [RS:1;dev1762:33025] zookeeper.ZKUtil(368): regionserver:33025-0x1523497b4040002, quorum=localhost:55120, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/dev1762.prn2.facebook.com,33025,1452581304563 2016-01-11 22:48:25,312 INFO [RS:0;dev1762:61792] regionserver.ReplicationSourceManager(236): Current list of replicators: [dev1762.prn2.facebook.com,61792,1452581304511, dev1762.prn2.facebook.com,33025,1452581304563] other RSs: [dev1762.prn2.facebook.com,61792,1452581304511, dev1762.prn2.facebook.com,33025,1452581304563] 2016-01-11 22:48:25,312 INFO [RS:1;dev1762:33025] regionserver.ReplicationSourceManager(236): Current list of replicators: [dev1762.prn2.facebook.com,61792,1452581304511, dev1762.prn2.facebook.com,33025,1452581304563] other RSs: [dev1762.prn2.facebook.com,61792,1452581304511, dev1762.prn2.facebook.com,33025,1452581304563] 2016-01-11 22:48:25,357 INFO [SplitLogWorker-dev1762:61792] regionserver.SplitLogWorker(134): SplitLogWorker dev1762.prn2.facebook.com,61792,1452581304511 starting 2016-01-11 22:48:25,357 INFO [SplitLogWorker-dev1762:33025] regionserver.SplitLogWorker(134): SplitLogWorker dev1762.prn2.facebook.com,33025,1452581304563 starting 2016-01-11 22:48:25,358 INFO [RS:0;dev1762:61792] regionserver.HeapMemoryManager(191): Starting HeapMemoryTuner chore. 2016-01-11 22:48:25,358 INFO [RS:1;dev1762:33025] regionserver.HeapMemoryManager(191): Starting HeapMemoryTuner chore. 2016-01-11 22:48:25,361 INFO [RS:1;dev1762:33025] regionserver.HRegionServer(1381): Serving as dev1762.prn2.facebook.com,33025,1452581304563, RpcServer on dev1762.prn2.facebook.com/10.35.11.29:33025, sessionid=0x1523497b4040002 2016-01-11 22:48:25,361 INFO [RS:0;dev1762:61792] regionserver.HRegionServer(1381): Serving as dev1762.prn2.facebook.com,61792,1452581304511, RpcServer on dev1762.prn2.facebook.com/10.35.11.29:61792, sessionid=0x1523497b4040001 2016-01-11 22:48:25,362 DEBUG [RS:0;dev1762:61792] procedure.RegionServerProcedureManagerHost(52): Procedure flush-table-proc is starting 2016-01-11 22:48:25,362 DEBUG [RS:1;dev1762:33025] procedure.RegionServerProcedureManagerHost(52): Procedure flush-table-proc is starting 2016-01-11 22:48:25,362 DEBUG [RS:0;dev1762:61792] flush.RegionServerFlushTableProcedureManager(103): Start region server flush procedure manager dev1762.prn2.facebook.com,61792,1452581304511 2016-01-11 22:48:25,362 DEBUG [RS:1;dev1762:33025] flush.RegionServerFlushTableProcedureManager(103): Start region server flush procedure manager dev1762.prn2.facebook.com,33025,1452581304563 2016-01-11 22:48:25,362 DEBUG [RS:0;dev1762:61792] procedure.ZKProcedureMemberRpcs(352): Starting procedure member 'dev1762.prn2.facebook.com,61792,1452581304511' 2016-01-11 22:48:25,362 DEBUG [RS:1;dev1762:33025] procedure.ZKProcedureMemberRpcs(352): Starting procedure member 'dev1762.prn2.facebook.com,33025,1452581304563' 2016-01-11 22:48:25,363 DEBUG [RS:0;dev1762:61792] procedure.ZKProcedureMemberRpcs(136): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2016-01-11 22:48:25,363 DEBUG [RS:1;dev1762:33025] procedure.ZKProcedureMemberRpcs(136): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2016-01-11 22:48:25,363 DEBUG [RS:0;dev1762:61792] procedure.ZKProcedureMemberRpcs(152): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2016-01-11 22:48:25,363 DEBUG [RS:1;dev1762:33025] procedure.ZKProcedureMemberRpcs(152): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2016-01-11 22:48:25,364 DEBUG [RS:0;dev1762:61792] procedure.RegionServerProcedureManagerHost(54): Procedure flush-table-proc is started 2016-01-11 22:48:25,364 DEBUG [RS:1;dev1762:33025] procedure.RegionServerProcedureManagerHost(54): Procedure flush-table-proc is started 2016-01-11 22:48:25,364 DEBUG [RS:0;dev1762:61792] procedure.RegionServerProcedureManagerHost(52): Procedure online-snapshot is starting 2016-01-11 22:48:25,364 DEBUG [RS:1;dev1762:33025] procedure.RegionServerProcedureManagerHost(52): Procedure online-snapshot is starting 2016-01-11 22:48:25,364 DEBUG [RS:0;dev1762:61792] snapshot.RegionServerSnapshotManager(124): Start Snapshot Manager dev1762.prn2.facebook.com,61792,1452581304511 2016-01-11 22:48:25,364 DEBUG [RS:1;dev1762:33025] snapshot.RegionServerSnapshotManager(124): Start Snapshot Manager dev1762.prn2.facebook.com,33025,1452581304563 2016-01-11 22:48:25,364 DEBUG [RS:0;dev1762:61792] procedure.ZKProcedureMemberRpcs(352): Starting procedure member 'dev1762.prn2.facebook.com,61792,1452581304511' 2016-01-11 22:48:25,364 DEBUG [RS:1;dev1762:33025] procedure.ZKProcedureMemberRpcs(352): Starting procedure member 'dev1762.prn2.facebook.com,33025,1452581304563' 2016-01-11 22:48:25,364 DEBUG [RS:0;dev1762:61792] procedure.ZKProcedureMemberRpcs(136): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2016-01-11 22:48:25,365 DEBUG [RS:1;dev1762:33025] procedure.ZKProcedureMemberRpcs(136): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2016-01-11 22:48:25,365 DEBUG [RS:0;dev1762:61792] procedure.ZKProcedureMemberRpcs(152): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2016-01-11 22:48:25,365 DEBUG [RS:1;dev1762:33025] procedure.ZKProcedureMemberRpcs(152): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2016-01-11 22:48:25,365 DEBUG [RS:0;dev1762:61792] procedure.RegionServerProcedureManagerHost(54): Procedure online-snapshot is started 2016-01-11 22:48:25,365 DEBUG [RS:1;dev1762:33025] procedure.RegionServerProcedureManagerHost(54): Procedure online-snapshot is started 2016-01-11 22:48:25,371 INFO [RS:0;dev1762:61792] quotas.RegionServerQuotaManager(53): Quota support disabled 2016-01-11 22:48:25,371 INFO [RS:1;dev1762:33025] quotas.RegionServerQuotaManager(53): Quota support disabled 2016-01-11 22:48:25,393 DEBUG [dev1762:49897.activeMasterManager] zookeeper.ZKUtil(627): master:49897-0x1523497b4040000, quorum=localhost:55120, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-01-11 22:48:25,594 DEBUG [dev1762:49897.activeMasterManager] zookeeper.ZKUtil(627): master:49897-0x1523497b4040000, quorum=localhost:55120, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-01-11 22:48:25,795 DEBUG [dev1762:49897.activeMasterManager] zookeeper.ZKUtil(627): master:49897-0x1523497b4040000, quorum=localhost:55120, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-01-11 22:48:25,996 DEBUG [dev1762:49897.activeMasterManager] zookeeper.ZKUtil(627): master:49897-0x1523497b4040000, quorum=localhost:55120, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-01-11 22:48:25,996 INFO [dev1762:49897.activeMasterManager] zookeeper.MetaTableLocator(525): Deleting hbase:meta region location in ZooKeeper 2016-01-11 22:48:26,059 DEBUG [dev1762:49897.activeMasterManager] zookeeper.RecoverableZooKeeper(188): Node /hbase/meta-region-server already deleted, retry=false 2016-01-11 22:48:26,065 DEBUG [dev1762:49897.activeMasterManager] balancer.RegionLocationFinder(269): HDFSBlocksDistribution not found in cache for region hbase:meta,,1.1588230740 2016-01-11 22:48:26,070 DEBUG [dev1762:49897.activeMasterManager] regionserver.HRegionFileSystem(201): No StoreFiles for: hdfs://localhost:59295/user/elliott/test-data/4f0ddc3b-9bf4-4ec6-b2ab-29e91003a033/data/hbase/meta/1588230740/info 2016-01-11 22:48:26,074 DEBUG [dev1762:49897.activeMasterManager] master.AssignmentManager(2454): No previous transition plan found (or ignoring an existing plan) for hbase:meta,,1.1588230740; generated random plan=hri=hbase:meta,,1.1588230740, src=, dest=dev1762.prn2.facebook.com,61792,1452581304511; 2 (online=2) available servers, forceNewPlan=false 2016-01-11 22:48:26,075 INFO [dev1762:49897.activeMasterManager] master.AssignmentManager(2100): Setting node as OFFLINED in ZooKeeper for region {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''} 2016-01-11 22:48:26,075 DEBUG [dev1762:49897.activeMasterManager] zookeeper.ZKAssign(205): master:49897-0x1523497b4040000, quorum=localhost:55120, baseZNode=/hbase Creating (or updating) unassigned node 1588230740 with OFFLINE state 2016-01-11 22:48:26,084 DEBUG [dev1762:49897.activeMasterManager] master.AssignmentManager(2117): Setting table hbase:meta to ENABLED state. 2016-01-11 22:48:26,084 INFO [dev1762:49897.activeMasterManager] zookeeper.ZKTableStateManager(100): Moving table hbase:meta state from null to ENABLED 2016-01-11 22:48:26,100 INFO [dev1762:49897.activeMasterManager] master.AssignmentManager(2132): Assigning hbase:meta,,1.1588230740 to dev1762.prn2.facebook.com,61792,1452581304511 2016-01-11 22:48:26,101 INFO [dev1762:49897.activeMasterManager] master.RegionStates(1109): Transition {1588230740 state=OFFLINE, ts=1452581306075, server=null} to {1588230740 state=PENDING_OPEN, ts=1452581306101, server=dev1762.prn2.facebook.com,61792,1452581304511} 2016-01-11 22:48:26,101 DEBUG [dev1762:49897.activeMasterManager] master.ServerManager(954): New admin connection to dev1762.prn2.facebook.com,61792,1452581304511 2016-01-11 22:48:26,116 DEBUG [dev1762:49897.activeMasterManager] ipc.RpcClientImpl$Connection(351): Use SIMPLE authentication for service AdminService, sasl=false 2016-01-11 22:48:26,116 DEBUG [dev1762:49897.activeMasterManager] ipc.RpcClientImpl$Connection(716): Connecting to dev1762.prn2.facebook.com/10.35.11.29:61792 2016-01-11 22:48:26,117 DEBUG [RpcServer.listener,port=61792] ipc.RpcServer$Listener(879): RpcServer.listener,port=61792: connection from 10.35.11.29:60588; # active connections: 1 2016-01-11 22:48:26,118 INFO [RpcServer.reader=1,bindAddress=dev1762.prn2.facebook.com,port=61792] ipc.RpcServer$Connection(1712): Auth successful for elliott (auth:SIMPLE) 2016-01-11 22:48:26,119 INFO [RpcServer.reader=1,bindAddress=dev1762.prn2.facebook.com,port=61792] ipc.RpcServer$Connection(1742): Connection from 10.35.11.29 port: 60588 with version info: version: "1.2.0" url: "git://dev1762.prn2.facebook.com/data/users/elliott/hbase" revision: "4b59f62c231a39fd4a824c0cbf4c2166c4ec2a76" user: "elliott" date: "Mon Jan 11 22:47:43 PST 2016" src_checksum: "58511d3ee667dd7df285c172560770ca" 2016-01-11 22:48:26,122 INFO [PriorityRpcServer.handler=1,queue=1,port=61792] regionserver.RSRpcServices(1557): Open hbase:meta,,1.1588230740 2016-01-11 22:48:26,126 DEBUG [RS_OPEN_META-dev1762:61792-0] zookeeper.ZKAssign(805): regionserver:61792-0x1523497b4040001, quorum=localhost:55120, baseZNode=/hbase Transitioning 1588230740 from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING 2016-01-11 22:48:26,131 INFO [dev1762:49897.activeMasterManager] master.ServerManager(687): AssignmentManager hasn't finished failover cleanup; waiting 2016-01-11 22:48:26,133 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(554): master:49897-0x1523497b4040000, quorum=localhost:55120, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/region-in-transition/1588230740 2016-01-11 22:48:26,133 DEBUG [RS_OPEN_META-dev1762:61792-0] zookeeper.ZKAssign(880): regionserver:61792-0x1523497b4040001, quorum=localhost:55120, baseZNode=/hbase Transitioned node 1588230740 from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING 2016-01-11 22:48:26,134 DEBUG [dev1762:49897.activeMasterManager] zookeeper.ZKUtil(627): master:49897-0x1523497b4040000, quorum=localhost:55120, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-01-11 22:48:26,134 INFO [RS_OPEN_META-dev1762:61792-0] wal.WALFactory(144): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.DefaultWALProvider 2016-01-11 22:48:26,134 DEBUG [dev1762:49897.activeMasterManager] zookeeper.ZKUtil(627): master:49897-0x1523497b4040000, quorum=localhost:55120, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-01-11 22:48:26,135 DEBUG [dev1762:49897.activeMasterManager] zookeeper.ZKUtil(627): master:49897-0x1523497b4040000, quorum=localhost:55120, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-01-11 22:48:26,135 DEBUG [AM.ZK.Worker-pool2-t1] master.AssignmentManager(956): Handling RS_ZK_REGION_OPENING, server=dev1762.prn2.facebook.com,61792,1452581304511, region=1588230740, current_state={1588230740 state=PENDING_OPEN, ts=1452581306101, server=dev1762.prn2.facebook.com,61792,1452581304511} 2016-01-11 22:48:26,136 INFO [AM.ZK.Worker-pool2-t1] master.RegionStates(1109): Transition {1588230740 state=PENDING_OPEN, ts=1452581306101, server=dev1762.prn2.facebook.com,61792,1452581304511} to {1588230740 state=OPENING, ts=1452581306136, server=dev1762.prn2.facebook.com,61792,1452581304511} 2016-01-11 22:48:26,138 DEBUG [dev1762:49897.activeMasterManager] zookeeper.ZKUtil(627): master:49897-0x1523497b4040000, quorum=localhost:55120, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-01-11 22:48:26,139 DEBUG [dev1762:49897.activeMasterManager] zookeeper.ZKUtil(627): master:49897-0x1523497b4040000, quorum=localhost:55120, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-01-11 22:48:26,140 DEBUG [dev1762:49897.activeMasterManager] zookeeper.ZKUtil(627): master:49897-0x1523497b4040000, quorum=localhost:55120, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-01-11 22:48:26,140 INFO [RS_OPEN_META-dev1762:61792-0] wal.FSHLog(529): WAL configuration: blocksize=128 MB, rollsize=121.60 MB, prefix=dev1762.prn2.facebook.com%2C61792%2C1452581304511..meta, suffix=.meta, logDir=hdfs://localhost:59295/user/elliott/test-data/4f0ddc3b-9bf4-4ec6-b2ab-29e91003a033/WALs/dev1762.prn2.facebook.com,61792,1452581304511, archiveDir=hdfs://localhost:59295/user/elliott/test-data/4f0ddc3b-9bf4-4ec6-b2ab-29e91003a033/oldWALs 2016-01-11 22:48:26,141 DEBUG [dev1762:49897.activeMasterManager] zookeeper.ZKUtil(627): master:49897-0x1523497b4040000, quorum=localhost:55120, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-01-11 22:48:26,141 DEBUG [dev1762:49897.activeMasterManager] zookeeper.ZKUtil(627): master:49897-0x1523497b4040000, quorum=localhost:55120, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-01-11 22:48:26,142 DEBUG [dev1762:49897.activeMasterManager] zookeeper.ZKUtil(627): master:49897-0x1523497b4040000, quorum=localhost:55120, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-01-11 22:48:26,143 DEBUG [dev1762:49897.activeMasterManager] zookeeper.ZKUtil(627): master:49897-0x1523497b4040000, quorum=localhost:55120, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-01-11 22:48:26,143 DEBUG [dev1762:49897.activeMasterManager] zookeeper.ZKUtil(627): master:49897-0x1523497b4040000, quorum=localhost:55120, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-01-11 22:48:26,144 DEBUG [dev1762:49897.activeMasterManager] zookeeper.ZKUtil(627): master:49897-0x1523497b4040000, quorum=localhost:55120, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-01-11 22:48:26,145 DEBUG [dev1762:49897.activeMasterManager] zookeeper.ZKUtil(627): master:49897-0x1523497b4040000, quorum=localhost:55120, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-01-11 22:48:26,146 DEBUG [dev1762:49897.activeMasterManager] zookeeper.ZKUtil(627): master:49897-0x1523497b4040000, quorum=localhost:55120, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-01-11 22:48:26,147 DEBUG [dev1762:49897.activeMasterManager] zookeeper.ZKUtil(627): master:49897-0x1523497b4040000, quorum=localhost:55120, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-01-11 22:48:26,148 DEBUG [dev1762:49897.activeMasterManager] zookeeper.ZKUtil(627): master:49897-0x1523497b4040000, quorum=localhost:55120, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-01-11 22:48:26,152 DEBUG [dev1762:49897.activeMasterManager] zookeeper.ZKUtil(627): master:49897-0x1523497b4040000, quorum=localhost:55120, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-01-11 22:48:26,154 DEBUG [dev1762:49897.activeMasterManager] zookeeper.ZKUtil(627): master:49897-0x1523497b4040000, quorum=localhost:55120, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-01-11 22:48:26,154 DEBUG [dev1762:49897.activeMasterManager] zookeeper.ZKUtil(627): master:49897-0x1523497b4040000, quorum=localhost:55120, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-01-11 22:48:26,154 INFO [RS_OPEN_META-dev1762:61792-0] wal.FSHLog(1397): Slow sync cost: 10 ms, current pipeline: [] 2016-01-11 22:48:26,155 INFO [RS_OPEN_META-dev1762:61792-0] wal.FSHLog(876): New WAL /user/elliott/test-data/4f0ddc3b-9bf4-4ec6-b2ab-29e91003a033/WALs/dev1762.prn2.facebook.com,61792,1452581304511/dev1762.prn2.facebook.com%2C61792%2C1452581304511..meta.1452581306140.meta 2016-01-11 22:48:26,155 DEBUG [dev1762:49897.activeMasterManager] zookeeper.ZKUtil(627): master:49897-0x1523497b4040000, quorum=localhost:55120, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-01-11 22:48:26,156 DEBUG [dev1762:49897.activeMasterManager] zookeeper.ZKUtil(627): master:49897-0x1523497b4040000, quorum=localhost:55120, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-01-11 22:48:26,156 DEBUG [dev1762:49897.activeMasterManager] zookeeper.ZKUtil(627): master:49897-0x1523497b4040000, quorum=localhost:55120, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-01-11 22:48:26,157 DEBUG [dev1762:49897.activeMasterManager] zookeeper.ZKUtil(627): master:49897-0x1523497b4040000, quorum=localhost:55120, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-01-11 22:48:26,157 DEBUG [dev1762:49897.activeMasterManager] zookeeper.ZKUtil(627): master:49897-0x1523497b4040000, quorum=localhost:55120, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-01-11 22:48:26,158 DEBUG [dev1762:49897.activeMasterManager] zookeeper.ZKUtil(627): master:49897-0x1523497b4040000, quorum=localhost:55120, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-01-11 22:48:26,158 DEBUG [RS_OPEN_META-dev1762:61792-0] regionserver.HRegion(6408): Opening region: {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''} 2016-01-11 22:48:26,159 DEBUG [dev1762:49897.activeMasterManager] zookeeper.ZKUtil(627): master:49897-0x1523497b4040000, quorum=localhost:55120, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-01-11 22:48:26,159 DEBUG [dev1762:49897.activeMasterManager] zookeeper.ZKUtil(627): master:49897-0x1523497b4040000, quorum=localhost:55120, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-01-11 22:48:26,160 DEBUG [dev1762:49897.activeMasterManager] zookeeper.ZKUtil(627): master:49897-0x1523497b4040000, quorum=localhost:55120, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-01-11 22:48:26,160 DEBUG [dev1762:49897.activeMasterManager] zookeeper.ZKUtil(627): master:49897-0x1523497b4040000, quorum=localhost:55120, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-01-11 22:48:26,161 DEBUG [dev1762:49897.activeMasterManager] zookeeper.ZKUtil(627): master:49897-0x1523497b4040000, quorum=localhost:55120, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-01-11 22:48:26,162 DEBUG [dev1762:49897.activeMasterManager] zookeeper.ZKUtil(627): master:49897-0x1523497b4040000, quorum=localhost:55120, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-01-11 22:48:26,162 DEBUG [dev1762:49897.activeMasterManager] zookeeper.ZKUtil(627): master:49897-0x1523497b4040000, quorum=localhost:55120, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-01-11 22:48:26,163 DEBUG [dev1762:49897.activeMasterManager] zookeeper.ZKUtil(627): master:49897-0x1523497b4040000, quorum=localhost:55120, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-01-11 22:48:26,164 DEBUG [dev1762:49897.activeMasterManager] zookeeper.ZKUtil(627): master:49897-0x1523497b4040000, quorum=localhost:55120, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-01-11 22:48:26,164 DEBUG [dev1762:49897.activeMasterManager] zookeeper.ZKUtil(627): master:49897-0x1523497b4040000, quorum=localhost:55120, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-01-11 22:48:26,165 DEBUG [dev1762:49897.activeMasterManager] zookeeper.ZKUtil(627): master:49897-0x1523497b4040000, quorum=localhost:55120, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-01-11 22:48:26,165 DEBUG [dev1762:49897.activeMasterManager] zookeeper.ZKUtil(627): master:49897-0x1523497b4040000, quorum=localhost:55120, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-01-11 22:48:26,166 DEBUG [dev1762:49897.activeMasterManager] zookeeper.ZKUtil(627): master:49897-0x1523497b4040000, quorum=localhost:55120, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-01-11 22:48:26,167 DEBUG [dev1762:49897.activeMasterManager] zookeeper.ZKUtil(627): master:49897-0x1523497b4040000, quorum=localhost:55120, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-01-11 22:48:26,167 DEBUG [dev1762:49897.activeMasterManager] zookeeper.ZKUtil(627): master:49897-0x1523497b4040000, quorum=localhost:55120, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-01-11 22:48:26,168 DEBUG [dev1762:49897.activeMasterManager] zookeeper.ZKUtil(627): master:49897-0x1523497b4040000, quorum=localhost:55120, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-01-11 22:48:26,169 DEBUG [dev1762:49897.activeMasterManager] zookeeper.ZKUtil(627): master:49897-0x1523497b4040000, quorum=localhost:55120, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-01-11 22:48:26,169 DEBUG [dev1762:49897.activeMasterManager] zookeeper.ZKUtil(627): master:49897-0x1523497b4040000, quorum=localhost:55120, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-01-11 22:48:26,170 DEBUG [dev1762:49897.activeMasterManager] zookeeper.ZKUtil(627): master:49897-0x1523497b4040000, quorum=localhost:55120, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-01-11 22:48:26,170 DEBUG [dev1762:49897.activeMasterManager] zookeeper.ZKUtil(627): master:49897-0x1523497b4040000, quorum=localhost:55120, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-01-11 22:48:26,171 DEBUG [dev1762:49897.activeMasterManager] zookeeper.ZKUtil(627): master:49897-0x1523497b4040000, quorum=localhost:55120, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-01-11 22:48:26,172 DEBUG [dev1762:49897.activeMasterManager] zookeeper.ZKUtil(627): master:49897-0x1523497b4040000, quorum=localhost:55120, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-01-11 22:48:26,173 DEBUG [dev1762:49897.activeMasterManager] zookeeper.ZKUtil(627): master:49897-0x1523497b4040000, quorum=localhost:55120, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-01-11 22:48:26,173 DEBUG [dev1762:49897.activeMasterManager] zookeeper.ZKUtil(627): master:49897-0x1523497b4040000, quorum=localhost:55120, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-01-11 22:48:26,174 DEBUG [dev1762:49897.activeMasterManager] zookeeper.ZKUtil(627): master:49897-0x1523497b4040000, quorum=localhost:55120, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-01-11 22:48:26,175 DEBUG [dev1762:49897.activeMasterManager] zookeeper.ZKUtil(627): master:49897-0x1523497b4040000, quorum=localhost:55120, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-01-11 22:48:26,175 DEBUG [dev1762:49897.activeMasterManager] zookeeper.ZKUtil(627): master:49897-0x1523497b4040000, quorum=localhost:55120, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-01-11 22:48:26,176 DEBUG [dev1762:49897.activeMasterManager] zookeeper.ZKUtil(627): master:49897-0x1523497b4040000, quorum=localhost:55120, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-01-11 22:48:26,176 DEBUG [dev1762:49897.activeMasterManager] zookeeper.ZKUtil(627): master:49897-0x1523497b4040000, quorum=localhost:55120, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-01-11 22:48:26,177 DEBUG [dev1762:49897.activeMasterManager] zookeeper.ZKUtil(627): master:49897-0x1523497b4040000, quorum=localhost:55120, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-01-11 22:48:26,178 DEBUG [dev1762:49897.activeMasterManager] zookeeper.ZKUtil(627): master:49897-0x1523497b4040000, quorum=localhost:55120, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-01-11 22:48:26,178 DEBUG [dev1762:49897.activeMasterManager] zookeeper.ZKUtil(627): master:49897-0x1523497b4040000, quorum=localhost:55120, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-01-11 22:48:26,179 DEBUG [dev1762:49897.activeMasterManager] zookeeper.ZKUtil(627): master:49897-0x1523497b4040000, quorum=localhost:55120, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-01-11 22:48:26,180 DEBUG [dev1762:49897.activeMasterManager] zookeeper.ZKUtil(627): master:49897-0x1523497b4040000, quorum=localhost:55120, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-01-11 22:48:26,180 DEBUG [dev1762:49897.activeMasterManager] zookeeper.ZKUtil(627): master:49897-0x1523497b4040000, quorum=localhost:55120, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-01-11 22:48:26,181 DEBUG [dev1762:49897.activeMasterManager] zookeeper.ZKUtil(627): master:49897-0x1523497b4040000, quorum=localhost:55120, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-01-11 22:48:26,181 DEBUG [dev1762:49897.activeMasterManager] zookeeper.ZKUtil(627): master:49897-0x1523497b4040000, quorum=localhost:55120, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-01-11 22:48:26,182 DEBUG [dev1762:49897.activeMasterManager] zookeeper.ZKUtil(627): master:49897-0x1523497b4040000, quorum=localhost:55120, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-01-11 22:48:26,183 DEBUG [dev1762:49897.activeMasterManager] zookeeper.ZKUtil(627): master:49897-0x1523497b4040000, quorum=localhost:55120, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-01-11 22:48:26,183 DEBUG [RS_OPEN_META-dev1762:61792-0] coprocessor.CoprocessorHost(182): Loading coprocessor class org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint with path null and priority 536870911 2016-01-11 22:48:26,183 DEBUG [dev1762:49897.activeMasterManager] zookeeper.ZKUtil(627): master:49897-0x1523497b4040000, quorum=localhost:55120, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-01-11 22:48:26,184 DEBUG [dev1762:49897.activeMasterManager] zookeeper.ZKUtil(627): master:49897-0x1523497b4040000, quorum=localhost:55120, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-01-11 22:48:26,184 DEBUG [dev1762:49897.activeMasterManager] zookeeper.ZKUtil(627): master:49897-0x1523497b4040000, quorum=localhost:55120, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-01-11 22:48:26,185 DEBUG [dev1762:49897.activeMasterManager] zookeeper.ZKUtil(627): master:49897-0x1523497b4040000, quorum=localhost:55120, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-01-11 22:48:26,186 DEBUG [dev1762:49897.activeMasterManager] zookeeper.ZKUtil(627): master:49897-0x1523497b4040000, quorum=localhost:55120, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-01-11 22:48:26,186 DEBUG [dev1762:49897.activeMasterManager] zookeeper.ZKUtil(627): master:49897-0x1523497b4040000, quorum=localhost:55120, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-01-11 22:48:26,187 DEBUG [dev1762:49897.activeMasterManager] zookeeper.ZKUtil(627): master:49897-0x1523497b4040000, quorum=localhost:55120, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-01-11 22:48:26,187 DEBUG [dev1762:49897.activeMasterManager] zookeeper.ZKUtil(627): master:49897-0x1523497b4040000, quorum=localhost:55120, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-01-11 22:48:26,188 DEBUG [dev1762:49897.activeMasterManager] zookeeper.ZKUtil(627): master:49897-0x1523497b4040000, quorum=localhost:55120, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-01-11 22:48:26,189 DEBUG [dev1762:49897.activeMasterManager] zookeeper.ZKUtil(627): master:49897-0x1523497b4040000, quorum=localhost:55120, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-01-11 22:48:26,189 DEBUG [RS_OPEN_META-dev1762:61792-0] regionserver.HRegion(7654): Registered coprocessor service: region=hbase:meta,,1 service=MultiRowMutationService 2016-01-11 22:48:26,189 DEBUG [dev1762:49897.activeMasterManager] zookeeper.ZKUtil(627): master:49897-0x1523497b4040000, quorum=localhost:55120, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-01-11 22:48:26,190 DEBUG [dev1762:49897.activeMasterManager] zookeeper.ZKUtil(627): master:49897-0x1523497b4040000, quorum=localhost:55120, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-01-11 22:48:26,190 DEBUG [dev1762:49897.activeMasterManager] zookeeper.ZKUtil(627): master:49897-0x1523497b4040000, quorum=localhost:55120, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-01-11 22:48:26,191 DEBUG [dev1762:49897.activeMasterManager] zookeeper.ZKUtil(627): master:49897-0x1523497b4040000, quorum=localhost:55120, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-01-11 22:48:26,191 DEBUG [dev1762:49897.activeMasterManager] zookeeper.ZKUtil(627): master:49897-0x1523497b4040000, quorum=localhost:55120, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-01-11 22:48:26,192 DEBUG [dev1762:49897.activeMasterManager] zookeeper.ZKUtil(627): master:49897-0x1523497b4040000, quorum=localhost:55120, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-01-11 22:48:26,193 DEBUG [dev1762:49897.activeMasterManager] zookeeper.ZKUtil(627): master:49897-0x1523497b4040000, quorum=localhost:55120, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-01-11 22:48:26,193 INFO [RS_OPEN_META-dev1762:61792-0] regionserver.RegionCoprocessorHost(367): Loaded coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint from HTD of hbase:meta successfully. 2016-01-11 22:48:26,193 DEBUG [dev1762:49897.activeMasterManager] zookeeper.ZKUtil(627): master:49897-0x1523497b4040000, quorum=localhost:55120, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-01-11 22:48:26,194 DEBUG [dev1762:49897.activeMasterManager] zookeeper.ZKUtil(627): master:49897-0x1523497b4040000, quorum=localhost:55120, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-01-11 22:48:26,194 DEBUG [dev1762:49897.activeMasterManager] zookeeper.ZKUtil(627): master:49897-0x1523497b4040000, quorum=localhost:55120, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-01-11 22:48:26,195 DEBUG [dev1762:49897.activeMasterManager] zookeeper.ZKUtil(627): master:49897-0x1523497b4040000, quorum=localhost:55120, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-01-11 22:48:26,195 DEBUG [dev1762:49897.activeMasterManager] zookeeper.ZKUtil(627): master:49897-0x1523497b4040000, quorum=localhost:55120, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-01-11 22:48:26,196 DEBUG [dev1762:49897.activeMasterManager] zookeeper.ZKUtil(627): master:49897-0x1523497b4040000, quorum=localhost:55120, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-01-11 22:48:26,196 DEBUG [dev1762:49897.activeMasterManager] zookeeper.ZKUtil(627): master:49897-0x1523497b4040000, quorum=localhost:55120, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-01-11 22:48:26,197 DEBUG [dev1762:49897.activeMasterManager] zookeeper.ZKUtil(627): master:49897-0x1523497b4040000, quorum=localhost:55120, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-01-11 22:48:26,197 DEBUG [dev1762:49897.activeMasterManager] zookeeper.ZKUtil(627): master:49897-0x1523497b4040000, quorum=localhost:55120, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-01-11 22:48:26,198 DEBUG [dev1762:49897.activeMasterManager] zookeeper.ZKUtil(627): master:49897-0x1523497b4040000, quorum=localhost:55120, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-01-11 22:48:26,198 DEBUG [dev1762:49897.activeMasterManager] zookeeper.ZKUtil(627): master:49897-0x1523497b4040000, quorum=localhost:55120, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-01-11 22:48:26,199 DEBUG [dev1762:49897.activeMasterManager] zookeeper.ZKUtil(627): master:49897-0x1523497b4040000, quorum=localhost:55120, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-01-11 22:48:26,199 DEBUG [RS_OPEN_META-dev1762:61792-0] regionserver.MetricsRegionSourceImpl(70): Creating new MetricsRegionSourceImpl for table meta 1588230740 2016-01-11 22:48:26,199 DEBUG [RS_OPEN_META-dev1762:61792-0] regionserver.HRegion(733): Instantiated hbase:meta,,1.1588230740 2016-01-11 22:48:26,199 DEBUG [dev1762:49897.activeMasterManager] zookeeper.ZKUtil(627): master:49897-0x1523497b4040000, quorum=localhost:55120, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-01-11 22:48:26,200 DEBUG [dev1762:49897.activeMasterManager] zookeeper.ZKUtil(627): master:49897-0x1523497b4040000, quorum=localhost:55120, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-01-11 22:48:26,201 DEBUG [dev1762:49897.activeMasterManager] zookeeper.ZKUtil(627): master:49897-0x1523497b4040000, quorum=localhost:55120, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-01-11 22:48:26,201 DEBUG [dev1762:49897.activeMasterManager] zookeeper.ZKUtil(627): master:49897-0x1523497b4040000, quorum=localhost:55120, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-01-11 22:48:26,202 DEBUG [dev1762:49897.activeMasterManager] zookeeper.ZKUtil(627): master:49897-0x1523497b4040000, quorum=localhost:55120, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-01-11 22:48:26,202 DEBUG [dev1762:49897.activeMasterManager] zookeeper.ZKUtil(627): master:49897-0x1523497b4040000, quorum=localhost:55120, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-01-11 22:48:26,203 DEBUG [dev1762:49897.activeMasterManager] zookeeper.ZKUtil(627): master:49897-0x1523497b4040000, quorum=localhost:55120, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-01-11 22:48:26,203 DEBUG [dev1762:49897.activeMasterManager] zookeeper.ZKUtil(627): master:49897-0x1523497b4040000, quorum=localhost:55120, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-01-11 22:48:26,204 DEBUG [dev1762:49897.activeMasterManager] zookeeper.ZKUtil(627): master:49897-0x1523497b4040000, quorum=localhost:55120, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-01-11 22:48:26,204 DEBUG [dev1762:49897.activeMasterManager] zookeeper.ZKUtil(627): master:49897-0x1523497b4040000, quorum=localhost:55120, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-01-11 22:48:26,205 INFO [StoreOpener-1588230740-1] hfile.CacheConfig(292): blockCache=LruBlockCache{blockCount=0, currentSize=1071840, freeSize=1042890464, maxSize=1043962304, heapSize=1071840, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2016-01-11 22:48:26,206 DEBUG [dev1762:49897.activeMasterManager] zookeeper.ZKUtil(627): master:49897-0x1523497b4040000, quorum=localhost:55120, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-01-11 22:48:26,206 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(104): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000 2016-01-11 22:48:26,206 DEBUG [dev1762:49897.activeMasterManager] zookeeper.ZKUtil(627): master:49897-0x1523497b4040000, quorum=localhost:55120, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-01-11 22:48:26,207 DEBUG [dev1762:49897.activeMasterManager] zookeeper.ZKUtil(627): master:49897-0x1523497b4040000, quorum=localhost:55120, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-01-11 22:48:26,207 DEBUG [StoreOpener-1588230740-1] regionserver.HRegionFileSystem(201): No StoreFiles for: hdfs://localhost:59295/user/elliott/test-data/4f0ddc3b-9bf4-4ec6-b2ab-29e91003a033/data/hbase/meta/1588230740/info 2016-01-11 22:48:26,207 DEBUG [dev1762:49897.activeMasterManager] zookeeper.ZKUtil(627): master:49897-0x1523497b4040000, quorum=localhost:55120, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-01-11 22:48:26,208 DEBUG [dev1762:49897.activeMasterManager] zookeeper.ZKUtil(627): master:49897-0x1523497b4040000, quorum=localhost:55120, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-01-11 22:48:26,209 DEBUG [dev1762:49897.activeMasterManager] zookeeper.ZKUtil(627): master:49897-0x1523497b4040000, quorum=localhost:55120, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-01-11 22:48:26,209 DEBUG [dev1762:49897.activeMasterManager] zookeeper.ZKUtil(627): master:49897-0x1523497b4040000, quorum=localhost:55120, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-01-11 22:48:26,210 DEBUG [dev1762:49897.activeMasterManager] zookeeper.ZKUtil(627): master:49897-0x1523497b4040000, quorum=localhost:55120, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-01-11 22:48:26,211 DEBUG [dev1762:49897.activeMasterManager] zookeeper.ZKUtil(627): master:49897-0x1523497b4040000, quorum=localhost:55120, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-01-11 22:48:26,211 DEBUG [RS_OPEN_META-dev1762:61792-0] regionserver.HRegion(3890): Found 0 recovered edits file(s) under hdfs://localhost:59295/user/elliott/test-data/4f0ddc3b-9bf4-4ec6-b2ab-29e91003a033/data/hbase/meta/1588230740 2016-01-11 22:48:26,211 DEBUG [dev1762:49897.activeMasterManager] zookeeper.ZKUtil(627): master:49897-0x1523497b4040000, quorum=localhost:55120, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-01-11 22:48:26,212 DEBUG [dev1762:49897.activeMasterManager] zookeeper.ZKUtil(627): master:49897-0x1523497b4040000, quorum=localhost:55120, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-01-11 22:48:26,212 DEBUG [dev1762:49897.activeMasterManager] zookeeper.ZKUtil(627): master:49897-0x1523497b4040000, quorum=localhost:55120, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-01-11 22:48:26,213 DEBUG [dev1762:49897.activeMasterManager] zookeeper.ZKUtil(627): master:49897-0x1523497b4040000, quorum=localhost:55120, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-01-11 22:48:26,213 DEBUG [dev1762:49897.activeMasterManager] zookeeper.ZKUtil(627): master:49897-0x1523497b4040000, quorum=localhost:55120, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-01-11 22:48:26,213 DEBUG [RS_OPEN_META-dev1762:61792-0] regionserver.FlushLargeStoresPolicy(56): hbase.hregion.percolumnfamilyflush.size.lower.bound is not specified, use global config(16777216) instead 2016-01-11 22:48:26,213 DEBUG [dev1762:49897.activeMasterManager] zookeeper.ZKUtil(627): master:49897-0x1523497b4040000, quorum=localhost:55120, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-01-11 22:48:26,214 DEBUG [dev1762:49897.activeMasterManager] zookeeper.ZKUtil(627): master:49897-0x1523497b4040000, quorum=localhost:55120, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-01-11 22:48:26,214 DEBUG [dev1762:49897.activeMasterManager] zookeeper.ZKUtil(627): master:49897-0x1523497b4040000, quorum=localhost:55120, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-01-11 22:48:26,215 DEBUG [dev1762:49897.activeMasterManager] zookeeper.ZKUtil(627): master:49897-0x1523497b4040000, quorum=localhost:55120, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-01-11 22:48:26,215 DEBUG [dev1762:49897.activeMasterManager] zookeeper.ZKUtil(627): master:49897-0x1523497b4040000, quorum=localhost:55120, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-01-11 22:48:26,216 DEBUG [dev1762:49897.activeMasterManager] zookeeper.ZKUtil(627): master:49897-0x1523497b4040000, quorum=localhost:55120, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-01-11 22:48:26,216 DEBUG [dev1762:49897.activeMasterManager] zookeeper.ZKUtil(627): master:49897-0x1523497b4040000, quorum=localhost:55120, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-01-11 22:48:26,217 DEBUG [dev1762:49897.activeMasterManager] zookeeper.ZKUtil(627): master:49897-0x1523497b4040000, quorum=localhost:55120, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-01-11 22:48:26,217 DEBUG [dev1762:49897.activeMasterManager] zookeeper.ZKUtil(627): master:49897-0x1523497b4040000, quorum=localhost:55120, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-01-11 22:48:26,218 DEBUG [dev1762:49897.activeMasterManager] zookeeper.ZKUtil(627): master:49897-0x1523497b4040000, quorum=localhost:55120, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-01-11 22:48:26,218 DEBUG [dev1762:49897.activeMasterManager] zookeeper.ZKUtil(627): master:49897-0x1523497b4040000, quorum=localhost:55120, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-01-11 22:48:26,219 DEBUG [RS_OPEN_META-dev1762:61792-0] wal.WALSplitter(729): Wrote region seqId=hdfs://localhost:59295/user/elliott/test-data/4f0ddc3b-9bf4-4ec6-b2ab-29e91003a033/data/hbase/meta/1588230740/recovered.edits/3.seqid to file, newSeqId=3, maxSeqId=2 2016-01-11 22:48:26,219 DEBUG [dev1762:49897.activeMasterManager] zookeeper.ZKUtil(627): master:49897-0x1523497b4040000, quorum=localhost:55120, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-01-11 22:48:26,219 DEBUG [dev1762:49897.activeMasterManager] zookeeper.ZKUtil(627): master:49897-0x1523497b4040000, quorum=localhost:55120, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-01-11 22:48:26,220 INFO [RS_OPEN_META-dev1762:61792-0] regionserver.HRegion(868): Onlined 1588230740; next sequenceid=3 2016-01-11 22:48:26,220 DEBUG [dev1762:49897.activeMasterManager] zookeeper.ZKUtil(627): master:49897-0x1523497b4040000, quorum=localhost:55120, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-01-11 22:48:26,220 DEBUG [dev1762:49897.activeMasterManager] zookeeper.ZKUtil(627): master:49897-0x1523497b4040000, quorum=localhost:55120, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-01-11 22:48:26,221 DEBUG [dev1762:49897.activeMasterManager] zookeeper.ZKUtil(627): master:49897-0x1523497b4040000, quorum=localhost:55120, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-01-11 22:48:26,221 DEBUG [dev1762:49897.activeMasterManager] zookeeper.ZKUtil(627): master:49897-0x1523497b4040000, quorum=localhost:55120, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-01-11 22:48:26,222 DEBUG [dev1762:49897.activeMasterManager] zookeeper.ZKUtil(627): master:49897-0x1523497b4040000, quorum=localhost:55120, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-01-11 22:48:26,222 DEBUG [dev1762:49897.activeMasterManager] zookeeper.ZKUtil(627): master:49897-0x1523497b4040000, quorum=localhost:55120, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-01-11 22:48:26,222 DEBUG [dev1762:49897.activeMasterManager] zookeeper.ZKUtil(627): master:49897-0x1523497b4040000, quorum=localhost:55120, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-01-11 22:48:26,223 DEBUG [dev1762:49897.activeMasterManager] zookeeper.ZKUtil(627): master:49897-0x1523497b4040000, quorum=localhost:55120, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-01-11 22:48:26,223 DEBUG [dev1762:49897.activeMasterManager] zookeeper.ZKUtil(627): master:49897-0x1523497b4040000, quorum=localhost:55120, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-01-11 22:48:26,224 DEBUG [dev1762:49897.activeMasterManager] zookeeper.ZKUtil(627): master:49897-0x1523497b4040000, quorum=localhost:55120, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-01-11 22:48:26,224 DEBUG [dev1762:49897.activeMasterManager] zookeeper.ZKUtil(627): master:49897-0x1523497b4040000, quorum=localhost:55120, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-01-11 22:48:26,225 DEBUG [dev1762:49897.activeMasterManager] zookeeper.ZKUtil(627): master:49897-0x1523497b4040000, quorum=localhost:55120, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-01-11 22:48:26,225 DEBUG [dev1762:49897.activeMasterManager] zookeeper.ZKUtil(627): master:49897-0x1523497b4040000, quorum=localhost:55120, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-01-11 22:48:26,226 DEBUG [dev1762:49897.activeMasterManager] zookeeper.ZKUtil(627): master:49897-0x1523497b4040000, quorum=localhost:55120, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-01-11 22:48:26,226 DEBUG [dev1762:49897.activeMasterManager] zookeeper.ZKUtil(627): master:49897-0x1523497b4040000, quorum=localhost:55120, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-01-11 22:48:26,227 DEBUG [dev1762:49897.activeMasterManager] zookeeper.ZKUtil(627): master:49897-0x1523497b4040000, quorum=localhost:55120, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-01-11 22:48:26,227 DEBUG [dev1762:49897.activeMasterManager] zookeeper.ZKUtil(627): master:49897-0x1523497b4040000, quorum=localhost:55120, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-01-11 22:48:26,228 DEBUG [dev1762:49897.activeMasterManager] zookeeper.ZKUtil(627): master:49897-0x1523497b4040000, quorum=localhost:55120, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-01-11 22:48:26,228 DEBUG [dev1762:49897.activeMasterManager] zookeeper.ZKUtil(627): master:49897-0x1523497b4040000, quorum=localhost:55120, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-01-11 22:48:26,229 DEBUG [dev1762:49897.activeMasterManager] zookeeper.ZKUtil(627): master:49897-0x1523497b4040000, quorum=localhost:55120, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-01-11 22:48:26,229 DEBUG [dev1762:49897.activeMasterManager] zookeeper.ZKUtil(627): master:49897-0x1523497b4040000, quorum=localhost:55120, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-01-11 22:48:26,230 DEBUG [dev1762:49897.activeMasterManager] zookeeper.ZKUtil(627): master:49897-0x1523497b4040000, quorum=localhost:55120, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-01-11 22:48:26,231 DEBUG [dev1762:49897.activeMasterManager] zookeeper.ZKUtil(627): master:49897-0x1523497b4040000, quorum=localhost:55120, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-01-11 22:48:26,231 DEBUG [dev1762:49897.activeMasterManager] zookeeper.ZKUtil(627): master:49897-0x1523497b4040000, quorum=localhost:55120, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-01-11 22:48:26,231 DEBUG [dev1762:49897.activeMasterManager] zookeeper.ZKUtil(627): master:49897-0x1523497b4040000, quorum=localhost:55120, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-01-11 22:48:26,232 DEBUG [dev1762:49897.activeMasterManager] zookeeper.ZKUtil(627): master:49897-0x1523497b4040000, quorum=localhost:55120, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-01-11 22:48:26,232 DEBUG [dev1762:49897.activeMasterManager] zookeeper.ZKUtil(627): master:49897-0x1523497b4040000, quorum=localhost:55120, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-01-11 22:48:26,233 DEBUG [dev1762:49897.activeMasterManager] zookeeper.ZKUtil(627): master:49897-0x1523497b4040000, quorum=localhost:55120, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-01-11 22:48:26,233 DEBUG [dev1762:49897.activeMasterManager] zookeeper.ZKUtil(627): master:49897-0x1523497b4040000, quorum=localhost:55120, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-01-11 22:48:26,234 DEBUG [dev1762:49897.activeMasterManager] zookeeper.ZKUtil(627): master:49897-0x1523497b4040000, quorum=localhost:55120, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-01-11 22:48:26,234 DEBUG [dev1762:49897.activeMasterManager] zookeeper.ZKUtil(627): master:49897-0x1523497b4040000, quorum=localhost:55120, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-01-11 22:48:26,234 DEBUG [dev1762:49897.activeMasterManager] zookeeper.ZKUtil(627): master:49897-0x1523497b4040000, quorum=localhost:55120, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-01-11 22:48:26,235 DEBUG [dev1762:49897.activeMasterManager] zookeeper.ZKUtil(627): master:49897-0x1523497b4040000, quorum=localhost:55120, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-01-11 22:48:26,235 DEBUG [dev1762:49897.activeMasterManager] zookeeper.ZKUtil(627): master:49897-0x1523497b4040000, quorum=localhost:55120, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-01-11 22:48:26,236 DEBUG [dev1762:49897.activeMasterManager] zookeeper.ZKUtil(627): master:49897-0x1523497b4040000, quorum=localhost:55120, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-01-11 22:48:26,236 DEBUG [dev1762:49897.activeMasterManager] zookeeper.ZKUtil(627): master:49897-0x1523497b4040000, quorum=localhost:55120, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-01-11 22:48:26,237 DEBUG [dev1762:49897.activeMasterManager] zookeeper.ZKUtil(627): master:49897-0x1523497b4040000, quorum=localhost:55120, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-01-11 22:48:26,238 DEBUG [dev1762:49897.activeMasterManager] zookeeper.ZKUtil(627): master:49897-0x1523497b4040000, quorum=localhost:55120, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-01-11 22:48:26,238 DEBUG [dev1762:49897.activeMasterManager] zookeeper.ZKUtil(627): master:49897-0x1523497b4040000, quorum=localhost:55120, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-01-11 22:48:26,239 DEBUG [dev1762:49897.activeMasterManager] zookeeper.ZKUtil(627): master:49897-0x1523497b4040000, quorum=localhost:55120, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-01-11 22:48:26,239 DEBUG [dev1762:49897.activeMasterManager] zookeeper.ZKUtil(627): master:49897-0x1523497b4040000, quorum=localhost:55120, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-01-11 22:48:26,239 DEBUG [dev1762:49897.activeMasterManager] zookeeper.ZKUtil(627): master:49897-0x1523497b4040000, quorum=localhost:55120, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-01-11 22:48:26,240 DEBUG [dev1762:49897.activeMasterManager] zookeeper.ZKUtil(627): master:49897-0x1523497b4040000, quorum=localhost:55120, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-01-11 22:48:26,240 DEBUG [dev1762:49897.activeMasterManager] zookeeper.ZKUtil(627): master:49897-0x1523497b4040000, quorum=localhost:55120, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-01-11 22:48:26,241 DEBUG [dev1762:49897.activeMasterManager] zookeeper.ZKUtil(627): master:49897-0x1523497b4040000, quorum=localhost:55120, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-01-11 22:48:26,243 DEBUG [dev1762:49897.activeMasterManager] zookeeper.ZKUtil(627): master:49897-0x1523497b4040000, quorum=localhost:55120, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-01-11 22:48:26,243 DEBUG [RS_OPEN_META-dev1762:61792-0] zookeeper.ZKAssign(643): regionserver:61792-0x1523497b4040001, quorum=localhost:55120, baseZNode=/hbase Attempting to retransition opening state of node 1588230740 2016-01-11 22:48:26,243 DEBUG [dev1762:49897.activeMasterManager] zookeeper.ZKUtil(627): master:49897-0x1523497b4040000, quorum=localhost:55120, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-01-11 22:48:26,244 DEBUG [dev1762:49897.activeMasterManager] zookeeper.ZKUtil(627): master:49897-0x1523497b4040000, quorum=localhost:55120, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-01-11 22:48:26,244 DEBUG [dev1762:49897.activeMasterManager] zookeeper.ZKUtil(627): master:49897-0x1523497b4040000, quorum=localhost:55120, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-01-11 22:48:26,244 DEBUG [dev1762:49897.activeMasterManager] zookeeper.ZKUtil(627): master:49897-0x1523497b4040000, quorum=localhost:55120, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-01-11 22:48:26,245 INFO [PostOpenDeployTasks:1588230740] regionserver.HRegionServer(1912): Post open deploy tasks for hbase:meta,,1.1588230740 2016-01-11 22:48:26,245 DEBUG [dev1762:49897.activeMasterManager] zookeeper.ZKUtil(627): master:49897-0x1523497b4040000, quorum=localhost:55120, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-01-11 22:48:26,245 INFO [PostOpenDeployTasks:1588230740] zookeeper.MetaTableLocator(439): Setting hbase:meta region location in ZooKeeper as dev1762.prn2.facebook.com,61792,1452581304511 2016-01-11 22:48:26,245 DEBUG [dev1762:49897.activeMasterManager] zookeeper.ZKUtil(627): master:49897-0x1523497b4040000, quorum=localhost:55120, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-01-11 22:48:26,246 DEBUG [dev1762:49897.activeMasterManager] zookeeper.ZKUtil(627): master:49897-0x1523497b4040000, quorum=localhost:55120, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-01-11 22:48:26,246 DEBUG [dev1762:49897.activeMasterManager] zookeeper.ZKUtil(627): master:49897-0x1523497b4040000, quorum=localhost:55120, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-01-11 22:48:26,247 DEBUG [dev1762:49897.activeMasterManager] zookeeper.ZKUtil(627): master:49897-0x1523497b4040000, quorum=localhost:55120, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-01-11 22:48:26,247 DEBUG [dev1762:49897.activeMasterManager] zookeeper.ZKUtil(627): master:49897-0x1523497b4040000, quorum=localhost:55120, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-01-11 22:48:26,247 DEBUG [dev1762:49897.activeMasterManager] zookeeper.ZKUtil(627): master:49897-0x1523497b4040000, quorum=localhost:55120, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-01-11 22:48:26,248 DEBUG [dev1762:49897.activeMasterManager] zookeeper.ZKUtil(627): master:49897-0x1523497b4040000, quorum=localhost:55120, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-01-11 22:48:26,248 DEBUG [dev1762:49897.activeMasterManager] zookeeper.ZKUtil(627): master:49897-0x1523497b4040000, quorum=localhost:55120, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-01-11 22:48:26,249 DEBUG [dev1762:49897.activeMasterManager] zookeeper.ZKUtil(627): master:49897-0x1523497b4040000, quorum=localhost:55120, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-01-11 22:48:26,249 DEBUG [dev1762:49897.activeMasterManager] zookeeper.ZKUtil(627): master:49897-0x1523497b4040000, quorum=localhost:55120, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-01-11 22:48:26,258 DEBUG [PostOpenDeployTasks:1588230740] zookeeper.MetaTableLocator(451): META region location doesn't exist, create it 2016-01-11 22:48:26,259 DEBUG [dev1762:49897.activeMasterManager] zookeeper.ZKUtil(627): master:49897-0x1523497b4040000, quorum=localhost:55120, baseZNode=/hbase Unable to get data of znode /hbase/meta-region-server because node does not exist (not an error) 2016-01-11 22:48:26,267 DEBUG [PostOpenDeployTasks:1588230740] regionserver.HRegionServer(1946): Finished post open deploy task for hbase:meta,,1.1588230740 2016-01-11 22:48:26,268 DEBUG [RS_OPEN_META-dev1762:61792-0] zookeeper.ZKAssign(805): regionserver:61792-0x1523497b4040001, quorum=localhost:55120, baseZNode=/hbase Transitioning 1588230740 from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENED 2016-01-11 22:48:26,275 DEBUG [RS_OPEN_META-dev1762:61792-0] zookeeper.ZKAssign(880): regionserver:61792-0x1523497b4040001, quorum=localhost:55120, baseZNode=/hbase Transitioned node 1588230740 from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENED 2016-01-11 22:48:26,275 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(554): master:49897-0x1523497b4040000, quorum=localhost:55120, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/region-in-transition/1588230740 2016-01-11 22:48:26,275 DEBUG [RS_OPEN_META-dev1762:61792-0] coordination.ZkOpenRegionCoordination(91): Transitioned 1588230740 to OPENED in zk on dev1762.prn2.facebook.com,61792,1452581304511 2016-01-11 22:48:26,276 DEBUG [RS_OPEN_META-dev1762:61792-0] handler.OpenRegionHandler(170): Opened hbase:meta,,1.1588230740 on dev1762.prn2.facebook.com,61792,1452581304511 2016-01-11 22:48:26,276 DEBUG [AM.ZK.Worker-pool2-t2] master.AssignmentManager(956): Handling RS_ZK_REGION_OPENED, server=dev1762.prn2.facebook.com,61792,1452581304511, region=1588230740, current_state={1588230740 state=OPENING, ts=1452581306136, server=dev1762.prn2.facebook.com,61792,1452581304511} 2016-01-11 22:48:26,276 INFO [AM.ZK.Worker-pool2-t2] master.RegionStates(1109): Transition {1588230740 state=OPENING, ts=1452581306136, server=dev1762.prn2.facebook.com,61792,1452581304511} to {1588230740 state=OPEN, ts=1452581306276, server=dev1762.prn2.facebook.com,61792,1452581304511} 2016-01-11 22:48:26,278 INFO [AM.ZK.Worker-pool2-t2] coordination.ZkOpenRegionCoordination(348): Handling OPENED of 1588230740 from dev1762.prn2.facebook.com,49897,1452581304150; deleting unassigned node 2016-01-11 22:48:26,283 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(554): master:49897-0x1523497b4040000, quorum=localhost:55120, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/region-in-transition/1588230740 2016-01-11 22:48:26,283 DEBUG [AM.ZK.Worker-pool2-t2] zookeeper.ZKAssign(479): master:49897-0x1523497b4040000, quorum=localhost:55120, baseZNode=/hbase Deleted unassigned node 1588230740 in expected state RS_ZK_REGION_OPENED 2016-01-11 22:48:26,285 DEBUG [AM.ZK.Worker-pool2-t3] master.AssignmentManager$4(1345): Znode hbase:meta,,1.1588230740 deleted, state: {1588230740 state=OPEN, ts=1452581306276, server=dev1762.prn2.facebook.com,61792,1452581304511} 2016-01-11 22:48:26,285 DEBUG [AM.ZK.Worker-pool2-t3] master.RegionStates(456): Onlined 1588230740 on dev1762.prn2.facebook.com,61792,1452581304511 2016-01-11 22:48:26,286 INFO [dev1762:49897.activeMasterManager] master.HMaster(944): hbase:meta with replicaId 0 assigned=1, rit=false, location=dev1762.prn2.facebook.com,61792,1452581304511 2016-01-11 22:48:26,324 DEBUG [hconnection-0x1db27713-shared--pool8-t1] ipc.RpcClientImpl$Connection(351): Use SIMPLE authentication for service ClientService, sasl=false 2016-01-11 22:48:26,324 DEBUG [hconnection-0x1db27713-shared--pool8-t1] ipc.RpcClientImpl$Connection(716): Connecting to dev1762.prn2.facebook.com/10.35.11.29:61792 2016-01-11 22:48:26,326 DEBUG [RpcServer.listener,port=61792] ipc.RpcServer$Listener(879): RpcServer.listener,port=61792: connection from 10.35.11.29:60591; # active connections: 2 2016-01-11 22:48:26,327 INFO [RpcServer.reader=2,bindAddress=dev1762.prn2.facebook.com,port=61792] ipc.RpcServer$Connection(1712): Auth successful for elliott (auth:SIMPLE) 2016-01-11 22:48:26,328 INFO [RpcServer.reader=2,bindAddress=dev1762.prn2.facebook.com,port=61792] ipc.RpcServer$Connection(1742): Connection from 10.35.11.29 port: 60591 with version info: version: "1.2.0" url: "git://dev1762.prn2.facebook.com/data/users/elliott/hbase" revision: "4b59f62c231a39fd4a824c0cbf4c2166c4ec2a76" user: "elliott" date: "Mon Jan 11 22:47:43 PST 2016" src_checksum: "58511d3ee667dd7df285c172560770ca" 2016-01-11 22:48:26,363 INFO [dev1762:49897.activeMasterManager] hbase.MetaMigrationConvertingToPB(165): hbase:meta doesn't have any entries to update. 2016-01-11 22:48:26,363 INFO [dev1762:49897.activeMasterManager] hbase.MetaMigrationConvertingToPB(131): META already up-to date with PB serialization 2016-01-11 22:48:26,371 DEBUG [dev1762:49897.activeMasterManager] zookeeper.ZKAssign(497): master:49897-0x1523497b4040000, quorum=localhost:55120, baseZNode=/hbase Deleting any existing unassigned nodes 2016-01-11 22:48:26,372 INFO [dev1762:49897.activeMasterManager] master.AssignmentManager(644): Clean cluster startup. Assigning user regions 2016-01-11 22:48:26,373 INFO [dev1762:49897.activeMasterManager] master.AssignmentManager(502): Joined the cluster in 9ms, failover=false 2016-01-11 22:48:26,376 DEBUG [region-location-0] regionserver.HRegionFileSystem(201): No StoreFiles for: hdfs://localhost:59295/user/elliott/test-data/4f0ddc3b-9bf4-4ec6-b2ab-29e91003a033/data/hbase/meta/1588230740/info 2016-01-11 22:48:26,381 INFO [dev1762:49897.activeMasterManager] master.TableNamespaceManager(90): Namespace table not found. Creating... 2016-01-11 22:48:26,607 DEBUG [dev1762:49897.activeMasterManager] procedure2.ProcedureExecutor(654): Procedure CreateTableProcedure (table=hbase:namespace) id=1 owner=elliott state=RUNNABLE:CREATE_TABLE_PRE_OPERATION added to the store. 2016-01-11 22:48:26,652 DEBUG [ProcedureExecutor-0] lock.ZKInterProcessLockBase(226): Acquired a lock for /hbase/table-lock/hbase:namespace/write-master:498970000000000 2016-01-11 22:48:26,836 INFO [IPC Server handler 1 on 59295] blockmanagement.BlockManager(2383): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:46850 is added to blk_1073741833_1009{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-2f4161bb-a789-4c3e-bb22-3dbf1babdf25:NORMAL|RBW], ReplicaUnderConstruction[[DISK]DS-940d5c3a-1db1-4402-9e4c-a19c2cc6af02:NORMAL|RBW]]} size 0 2016-01-11 22:48:26,837 INFO [IPC Server handler 0 on 59295] blockmanagement.BlockManager(2383): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:38785 is added to blk_1073741833_1009{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-2f4161bb-a789-4c3e-bb22-3dbf1babdf25:NORMAL|RBW], ReplicaUnderConstruction[[DISK]DS-940d5c3a-1db1-4402-9e4c-a19c2cc6af02:NORMAL|RBW]]} size 0 2016-01-11 22:48:26,840 DEBUG [ProcedureExecutor-0] util.FSTableDescriptors(656): Wrote descriptor into: hdfs://localhost:59295/user/elliott/test-data/4f0ddc3b-9bf4-4ec6-b2ab-29e91003a033/.tmp/data/hbase/namespace/.tabledesc/.tableinfo.0000000001 2016-01-11 22:48:26,843 INFO [RegionOpenAndInitThread-hbase:namespace-1] regionserver.HRegion(6219): creating HRegion hbase:namespace HTD == 'hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', VERSIONS => '10', IN_MEMORY => 'true', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', COMPRESSION => 'NONE', CACHE_DATA_IN_L1 => 'true', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} RootDir = hdfs://localhost:59295/user/elliott/test-data/4f0ddc3b-9bf4-4ec6-b2ab-29e91003a033/.tmp Table name == hbase:namespace 2016-01-11 22:48:26,860 INFO [IPC Server handler 2 on 59295] blockmanagement.BlockManager(2383): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:38785 is added to blk_1073741834_1010{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-4449a44e-3e96-4db5-b41c-f22c79a44f57:NORMAL|RBW], ReplicaUnderConstruction[[DISK]DS-e3b160d5-0860-46e6-8012-66df73514c0f:NORMAL|RBW]]} size 0 2016-01-11 22:48:26,862 INFO [IPC Server handler 3 on 59295] blockmanagement.BlockManager(2383): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:46850 is added to blk_1073741834_1010{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-4449a44e-3e96-4db5-b41c-f22c79a44f57:NORMAL|RBW], ReplicaUnderConstruction[[DISK]DS-e3b160d5-0860-46e6-8012-66df73514c0f:NORMAL|RBW]]} size 0 2016-01-11 22:48:26,863 DEBUG [RegionOpenAndInitThread-hbase:namespace-1] regionserver.HRegion(733): Instantiated hbase:namespace,,1452581306381.8fdaad26859458841be03061163dcfe7. 2016-01-11 22:48:26,864 DEBUG [RegionOpenAndInitThread-hbase:namespace-1] regionserver.HRegion(1389): Closing hbase:namespace,,1452581306381.8fdaad26859458841be03061163dcfe7.: disabling compactions & flushes 2016-01-11 22:48:26,864 DEBUG [RegionOpenAndInitThread-hbase:namespace-1] regionserver.HRegion(1416): Updates disabled for region hbase:namespace,,1452581306381.8fdaad26859458841be03061163dcfe7. 2016-01-11 22:48:26,864 INFO [RegionOpenAndInitThread-hbase:namespace-1] regionserver.HRegion(1519): Closed hbase:namespace,,1452581306381.8fdaad26859458841be03061163dcfe7. 2016-01-11 22:48:27,061 INFO [ProcedureExecutor-0] hbase.MetaTableAccessor(1199): Added 1 2016-01-11 22:48:27,213 INFO [ProcedureExecutor-0] zookeeper.ZKTableStateManager(100): Moving table hbase:namespace state from null to ENABLING 2016-01-11 22:48:27,235 DEBUG [ProcedureExecutor-0] balancer.RegionLocationFinder(269): HDFSBlocksDistribution not found in cache for region hbase:namespace,,1452581306381.8fdaad26859458841be03061163dcfe7. 2016-01-11 22:48:27,242 DEBUG [ProcedureExecutor-0] regionserver.HRegionFileSystem(201): No StoreFiles for: hdfs://localhost:59295/user/elliott/test-data/4f0ddc3b-9bf4-4ec6-b2ab-29e91003a033/data/hbase/namespace/8fdaad26859458841be03061163dcfe7/info 2016-01-11 22:48:27,242 INFO [ProcedureExecutor-0] master.AssignmentManager(1613): Assigning 1 region(s) to dev1762.prn2.facebook.com,61792,1452581304511 2016-01-11 22:48:27,244 DEBUG [ProcedureExecutor-0] zookeeper.ZKAssign(174): master:49897-0x1523497b4040000, quorum=localhost:55120, baseZNode=/hbase Async create of unassigned node 8fdaad26859458841be03061163dcfe7 with OFFLINE state 2016-01-11 22:48:27,250 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(554): master:49897-0x1523497b4040000, quorum=localhost:55120, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/region-in-transition 2016-01-11 22:48:27,252 DEBUG [main-EventThread] master.OfflineCallback(69): rs={8fdaad26859458841be03061163dcfe7 state=OFFLINE, ts=1452581307235, server=null}, server=dev1762.prn2.facebook.com,61792,1452581304511 2016-01-11 22:48:27,253 DEBUG [main-EventThread] master.OfflineCallback$ExistCallback(106): rs={8fdaad26859458841be03061163dcfe7 state=OFFLINE, ts=1452581307235, server=null}, server=dev1762.prn2.facebook.com,61792,1452581304511 2016-01-11 22:48:27,254 DEBUG [ProcedureExecutor-0] master.AssignmentManager(1666): dev1762.prn2.facebook.com,61792,1452581304511 unassigned znodes=1 of total=1; oldCounter=0 2016-01-11 22:48:27,255 INFO [ProcedureExecutor-0] master.RegionStates(1109): Transition {8fdaad26859458841be03061163dcfe7 state=OFFLINE, ts=1452581307244, server=null} to {8fdaad26859458841be03061163dcfe7 state=PENDING_OPEN, ts=1452581307255, server=dev1762.prn2.facebook.com,61792,1452581304511} 2016-01-11 22:48:27,258 INFO [PriorityRpcServer.handler=4,queue=0,port=61792] regionserver.RSRpcServices(1557): Open hbase:namespace,,1452581306381.8fdaad26859458841be03061163dcfe7. 2016-01-11 22:48:27,263 DEBUG [RS_OPEN_REGION-dev1762:61792-0] zookeeper.ZKAssign(805): regionserver:61792-0x1523497b4040001, quorum=localhost:55120, baseZNode=/hbase Transitioning 8fdaad26859458841be03061163dcfe7 from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING 2016-01-11 22:48:27,264 DEBUG [ProcedureExecutor-0] master.AssignmentManager(1808): Bulk assigning done for dev1762.prn2.facebook.com,61792,1452581304511 2016-01-11 22:48:27,264 INFO [ProcedureExecutor-0] zookeeper.ZKTableStateManager(100): Moving table hbase:namespace state from ENABLING to ENABLED 2016-01-11 22:48:27,267 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(554): master:49897-0x1523497b4040000, quorum=localhost:55120, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/region-in-transition/8fdaad26859458841be03061163dcfe7 2016-01-11 22:48:27,267 DEBUG [RS_OPEN_REGION-dev1762:61792-0] zookeeper.ZKAssign(880): regionserver:61792-0x1523497b4040001, quorum=localhost:55120, baseZNode=/hbase Transitioned node 8fdaad26859458841be03061163dcfe7 from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING 2016-01-11 22:48:27,267 DEBUG [RS_OPEN_REGION-dev1762:61792-0] regionserver.HRegion(6408): Opening region: {ENCODED => 8fdaad26859458841be03061163dcfe7, NAME => 'hbase:namespace,,1452581306381.8fdaad26859458841be03061163dcfe7.', STARTKEY => '', ENDKEY => ''} 2016-01-11 22:48:27,268 DEBUG [RS_OPEN_REGION-dev1762:61792-0] regionserver.MetricsRegionSourceImpl(70): Creating new MetricsRegionSourceImpl for table namespace 8fdaad26859458841be03061163dcfe7 2016-01-11 22:48:27,268 DEBUG [RS_OPEN_REGION-dev1762:61792-0] regionserver.HRegion(733): Instantiated hbase:namespace,,1452581306381.8fdaad26859458841be03061163dcfe7. 2016-01-11 22:48:27,272 INFO [StoreOpener-8fdaad26859458841be03061163dcfe7-1] hfile.CacheConfig(292): blockCache=LruBlockCache{blockCount=0, currentSize=1071840, freeSize=1042890464, maxSize=1043962304, heapSize=1071840, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2016-01-11 22:48:27,272 INFO [StoreOpener-8fdaad26859458841be03061163dcfe7-1] compactions.CompactionConfiguration(104): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000 2016-01-11 22:48:27,273 DEBUG [StoreOpener-8fdaad26859458841be03061163dcfe7-1] regionserver.HRegionFileSystem(201): No StoreFiles for: hdfs://localhost:59295/user/elliott/test-data/4f0ddc3b-9bf4-4ec6-b2ab-29e91003a033/data/hbase/namespace/8fdaad26859458841be03061163dcfe7/info 2016-01-11 22:48:27,274 DEBUG [RS_OPEN_REGION-dev1762:61792-0] regionserver.HRegion(3890): Found 0 recovered edits file(s) under hdfs://localhost:59295/user/elliott/test-data/4f0ddc3b-9bf4-4ec6-b2ab-29e91003a033/data/hbase/namespace/8fdaad26859458841be03061163dcfe7 2016-01-11 22:48:27,275 DEBUG [AM.ZK.Worker-pool2-t5] master.AssignmentManager(956): Handling RS_ZK_REGION_OPENING, server=dev1762.prn2.facebook.com,61792,1452581304511, region=8fdaad26859458841be03061163dcfe7, current_state={8fdaad26859458841be03061163dcfe7 state=PENDING_OPEN, ts=1452581307255, server=dev1762.prn2.facebook.com,61792,1452581304511} 2016-01-11 22:48:27,275 INFO [AM.ZK.Worker-pool2-t5] master.RegionStates(1109): Transition {8fdaad26859458841be03061163dcfe7 state=PENDING_OPEN, ts=1452581307255, server=dev1762.prn2.facebook.com,61792,1452581304511} to {8fdaad26859458841be03061163dcfe7 state=OPENING, ts=1452581307275, server=dev1762.prn2.facebook.com,61792,1452581304511} 2016-01-11 22:48:27,276 DEBUG [RS_OPEN_REGION-dev1762:61792-0] regionserver.FlushLargeStoresPolicy(56): hbase.hregion.percolumnfamilyflush.size.lower.bound is not specified, use global config(16777216) instead 2016-01-11 22:48:27,280 DEBUG [RS_OPEN_REGION-dev1762:61792-0] wal.WALSplitter(729): Wrote region seqId=hdfs://localhost:59295/user/elliott/test-data/4f0ddc3b-9bf4-4ec6-b2ab-29e91003a033/data/hbase/namespace/8fdaad26859458841be03061163dcfe7/recovered.edits/2.seqid to file, newSeqId=2, maxSeqId=0 2016-01-11 22:48:27,280 INFO [RS_OPEN_REGION-dev1762:61792-0] regionserver.HRegion(868): Onlined 8fdaad26859458841be03061163dcfe7; next sequenceid=2 2016-01-11 22:48:27,284 DEBUG [RS_OPEN_REGION-dev1762:61792-0] zookeeper.ZKAssign(643): regionserver:61792-0x1523497b4040001, quorum=localhost:55120, baseZNode=/hbase Attempting to retransition opening state of node 8fdaad26859458841be03061163dcfe7 2016-01-11 22:48:27,285 INFO [PostOpenDeployTasks:8fdaad26859458841be03061163dcfe7] regionserver.HRegionServer(1912): Post open deploy tasks for hbase:namespace,,1452581306381.8fdaad26859458841be03061163dcfe7. 2016-01-11 22:48:27,292 INFO [PostOpenDeployTasks:8fdaad26859458841be03061163dcfe7] hbase.MetaTableAccessor(1394): Updated row hbase:namespace,,1452581306381.8fdaad26859458841be03061163dcfe7. with server=dev1762.prn2.facebook.com,61792,1452581304511 2016-01-11 22:48:27,292 DEBUG [PostOpenDeployTasks:8fdaad26859458841be03061163dcfe7] regionserver.HRegionServer(1946): Finished post open deploy task for hbase:namespace,,1452581306381.8fdaad26859458841be03061163dcfe7. 2016-01-11 22:48:27,292 DEBUG [RS_OPEN_REGION-dev1762:61792-0] zookeeper.ZKAssign(805): regionserver:61792-0x1523497b4040001, quorum=localhost:55120, baseZNode=/hbase Transitioning 8fdaad26859458841be03061163dcfe7 from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENED 2016-01-11 22:48:27,300 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(554): master:49897-0x1523497b4040000, quorum=localhost:55120, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/region-in-transition/8fdaad26859458841be03061163dcfe7 2016-01-11 22:48:27,300 DEBUG [RS_OPEN_REGION-dev1762:61792-0] zookeeper.ZKAssign(880): regionserver:61792-0x1523497b4040001, quorum=localhost:55120, baseZNode=/hbase Transitioned node 8fdaad26859458841be03061163dcfe7 from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENED 2016-01-11 22:48:27,300 DEBUG [RS_OPEN_REGION-dev1762:61792-0] coordination.ZkOpenRegionCoordination(91): Transitioned 8fdaad26859458841be03061163dcfe7 to OPENED in zk on dev1762.prn2.facebook.com,61792,1452581304511 2016-01-11 22:48:27,300 DEBUG [RS_OPEN_REGION-dev1762:61792-0] handler.OpenRegionHandler(170): Opened hbase:namespace,,1452581306381.8fdaad26859458841be03061163dcfe7. on dev1762.prn2.facebook.com,61792,1452581304511 2016-01-11 22:48:27,301 DEBUG [AM.ZK.Worker-pool2-t6] master.AssignmentManager(956): Handling RS_ZK_REGION_OPENED, server=dev1762.prn2.facebook.com,61792,1452581304511, region=8fdaad26859458841be03061163dcfe7, current_state={8fdaad26859458841be03061163dcfe7 state=OPENING, ts=1452581307275, server=dev1762.prn2.facebook.com,61792,1452581304511} 2016-01-11 22:48:27,301 INFO [AM.ZK.Worker-pool2-t6] master.RegionStates(1109): Transition {8fdaad26859458841be03061163dcfe7 state=OPENING, ts=1452581307275, server=dev1762.prn2.facebook.com,61792,1452581304511} to {8fdaad26859458841be03061163dcfe7 state=OPEN, ts=1452581307301, server=dev1762.prn2.facebook.com,61792,1452581304511} 2016-01-11 22:48:27,301 DEBUG [AM.ZK.Worker-pool2-t6] coordination.ZkOpenRegionCoordination(350): Handling OPENED of 8fdaad26859458841be03061163dcfe7 from dev1762.prn2.facebook.com,49897,1452581304150; deleting unassigned node 2016-01-11 22:48:27,308 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(554): master:49897-0x1523497b4040000, quorum=localhost:55120, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/region-in-transition/8fdaad26859458841be03061163dcfe7 2016-01-11 22:48:27,308 DEBUG [AM.ZK.Worker-pool2-t6] zookeeper.ZKAssign(479): master:49897-0x1523497b4040000, quorum=localhost:55120, baseZNode=/hbase Deleted unassigned node 8fdaad26859458841be03061163dcfe7 in expected state RS_ZK_REGION_OPENED 2016-01-11 22:48:27,309 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(554): master:49897-0x1523497b4040000, quorum=localhost:55120, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/region-in-transition 2016-01-11 22:48:27,310 DEBUG [AM.ZK.Worker-pool2-t8] master.AssignmentManager$4(1345): Znode hbase:namespace,,1452581306381.8fdaad26859458841be03061163dcfe7. deleted, state: {8fdaad26859458841be03061163dcfe7 state=OPEN, ts=1452581307301, server=dev1762.prn2.facebook.com,61792,1452581304511} 2016-01-11 22:48:27,310 DEBUG [AM.ZK.Worker-pool2-t8] master.RegionStates(456): Onlined 8fdaad26859458841be03061163dcfe7 on dev1762.prn2.facebook.com,61792,1452581304511 2016-01-11 22:48:27,312 DEBUG [dev1762:49897.activeMasterManager] zookeeper.ZKUtil(370): master:49897-0x1523497b4040000, quorum=localhost:55120, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/namespace 2016-01-11 22:48:27,338 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(554): master:49897-0x1523497b4040000, quorum=localhost:55120, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/namespace 2016-01-11 22:48:27,383 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(554): master:49897-0x1523497b4040000, quorum=localhost:55120, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2016-01-11 22:48:27,436 DEBUG [main-EventThread] hbase.ZKNamespaceManager(196): Updating namespace cache from node default with data: \x0A\x07default 2016-01-11 22:48:27,458 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(554): master:49897-0x1523497b4040000, quorum=localhost:55120, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2016-01-11 22:48:27,468 DEBUG [main-EventThread] hbase.ZKNamespaceManager(196): Updating namespace cache from node default with data: \x0A\x07default 2016-01-11 22:48:27,468 DEBUG [main-EventThread] hbase.ZKNamespaceManager(196): Updating namespace cache from node hbase with data: \x0A\x05hbase 2016-01-11 22:48:27,475 DEBUG [dev1762:49897.activeMasterManager] zookeeper.RecoverableZooKeeper(594): Node /hbase/namespace/default already exists 2016-01-11 22:48:27,483 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(554): master:49897-0x1523497b4040000, quorum=localhost:55120, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/default 2016-01-11 22:48:27,492 DEBUG [dev1762:49897.activeMasterManager] zookeeper.RecoverableZooKeeper(594): Node /hbase/namespace/hbase already exists 2016-01-11 22:48:27,500 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(554): master:49897-0x1523497b4040000, quorum=localhost:55120, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/hbase 2016-01-11 22:48:27,500 INFO [dev1762:49897.activeMasterManager] master.HMaster(775): Master has completed initialization 2016-01-11 22:48:27,502 INFO [dev1762:49897.activeMasterManager] quotas.MasterQuotaManager(62): Quota support disabled 2016-01-11 22:48:27,591 INFO [main] zookeeper.RecoverableZooKeeper(120): Process identifier=hconnection-0x173373b4 connecting to ZooKeeper ensemble=localhost:55120 2016-01-11 22:48:27,617 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(554): hconnection-0x173373b40x0, quorum=localhost:55120, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2016-01-11 22:48:27,618 DEBUG [main] ipc.AbstractRpcClient(115): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@40d10481, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2016-01-11 22:48:27,619 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(638): hconnection-0x173373b4-0x1523497b4040008 connected 2016-01-11 22:48:27,621 DEBUG [htable-pool12-t1] ipc.RpcClientImpl$Connection(351): Use SIMPLE authentication for service ClientService, sasl=false 2016-01-11 22:48:27,621 DEBUG [htable-pool12-t1] ipc.RpcClientImpl$Connection(716): Connecting to dev1762.prn2.facebook.com/10.35.11.29:61792 2016-01-11 22:48:27,622 DEBUG [RpcServer.listener,port=61792] ipc.RpcServer$Listener(879): RpcServer.listener,port=61792: connection from 10.35.11.29:60602; # active connections: 3 2016-01-11 22:48:27,623 INFO [RpcServer.reader=0,bindAddress=dev1762.prn2.facebook.com,port=61792] ipc.RpcServer$Connection(1712): Auth successful for elliott (auth:SIMPLE) 2016-01-11 22:48:27,623 INFO [RpcServer.reader=0,bindAddress=dev1762.prn2.facebook.com,port=61792] ipc.RpcServer$Connection(1742): Connection from 10.35.11.29 port: 60602 with version info: version: "1.2.0" url: "git://dev1762.prn2.facebook.com/data/users/elliott/hbase" revision: "4b59f62c231a39fd4a824c0cbf4c2166c4ec2a76" user: "elliott" date: "Mon Jan 11 22:47:43 PST 2016" src_checksum: "58511d3ee667dd7df285c172560770ca" 2016-01-11 22:48:27,628 INFO [main] client.ConnectionManager$HConnectionImplementation(1712): Closing zookeeper sessionid=0x1523497b4040008 2016-01-11 22:48:27,633 DEBUG [main] ipc.RpcClientImpl(1147): Stopping rpc client 2016-01-11 22:48:27,635 DEBUG [RpcServer.reader=0,bindAddress=dev1762.prn2.facebook.com,port=61792] ipc.RpcServer$Listener(911): RpcServer.listener,port=61792: DISCONNECTING client 10.35.11.29:60602 because read count=-1. Number of active connections: 3 2016-01-11 22:48:27,661 INFO [main] zookeeper.RecoverableZooKeeper(120): Process identifier=hconnection-0x16a9a4f1 connecting to ZooKeeper ensemble=localhost:55120 2016-01-11 22:48:27,667 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(554): hconnection-0x16a9a4f10x0, quorum=localhost:55120, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2016-01-11 22:48:27,668 DEBUG [main] ipc.AbstractRpcClient(115): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1f1cae23, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2016-01-11 22:48:27,668 INFO [main] hbase.HBaseTestingUtility(1043): Minicluster is up 2016-01-11 22:48:27,669 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(638): hconnection-0x16a9a4f1-0x1523497b4040009 connected 2016-01-11 22:48:27,669 INFO [main] hbase.HBaseTestingUtility(1211): The hbase.fs.tmp.dir is set to /user/elliott/hbase-staging 2016-01-11 22:48:27,674 DEBUG [main] ipc.RpcClientImpl$Connection(351): Use SIMPLE authentication for service MasterService, sasl=false 2016-01-11 22:48:27,674 DEBUG [main] ipc.RpcClientImpl$Connection(716): Connecting to dev1762.prn2.facebook.com/10.35.11.29:49897 2016-01-11 22:48:27,675 DEBUG [RpcServer.listener,port=49897] ipc.RpcServer$Listener(879): RpcServer.listener,port=49897: connection from 10.35.11.29:47737; # active connections: 3 2016-01-11 22:48:27,676 INFO [RpcServer.reader=0,bindAddress=dev1762.prn2.facebook.com,port=49897] ipc.RpcServer$Connection(1712): Auth successful for elliott (auth:SIMPLE) 2016-01-11 22:48:27,676 INFO [RpcServer.reader=0,bindAddress=dev1762.prn2.facebook.com,port=49897] ipc.RpcServer$Connection(1742): Connection from 10.35.11.29 port: 47737 with version info: version: "1.2.0" url: "git://dev1762.prn2.facebook.com/data/users/elliott/hbase" revision: "4b59f62c231a39fd4a824c0cbf4c2166c4ec2a76" user: "elliott" date: "Mon Jan 11 22:47:43 PST 2016" src_checksum: "58511d3ee667dd7df285c172560770ca" 2016-01-11 22:48:27,682 INFO [B.defaultRpcServer.handler=3,queue=0,port=49897] master.HMaster(2428): Client=elliott//10.35.11.29 creating {NAME => 'TestSplitWalDataLoss'} 2016-01-11 22:48:27,692 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(554): master:49897-0x1523497b4040000, quorum=localhost:55120, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2016-01-11 22:48:27,701 DEBUG [main-EventThread] hbase.ZKNamespaceManager(196): Updating namespace cache from node default with data: \x0A\x07default 2016-01-11 22:48:27,701 DEBUG [main-EventThread] hbase.ZKNamespaceManager(196): Updating namespace cache from node hbase with data: \x0A\x05hbase 2016-01-11 22:48:27,701 DEBUG [main-EventThread] hbase.ZKNamespaceManager(196): Updating namespace cache from node TestSplitWalDataLoss with data: \x0A\x14TestSplitWalDataLoss 2016-01-11 22:48:27,706 INFO [B.defaultRpcServer.handler=1,queue=0,port=49897] master.HMaster(1481): Client=elliott//10.35.11.29 create 'TestSplitWalDataLoss:dataloss', {NAME => 'f', BLOOMFILTER => 'ROW', VERSIONS => '1', IN_MEMORY => 'false', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', COMPRESSION => 'NONE', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2016-01-11 22:48:27,774 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=49897] procedure2.ProcedureExecutor(654): Procedure CreateTableProcedure (table=TestSplitWalDataLoss:dataloss) id=2 owner=elliott state=RUNNABLE:CREATE_TABLE_PRE_OPERATION added to the store. 2016-01-11 22:48:27,783 DEBUG [ProcedureExecutor-0] lock.ZKInterProcessLockBase(328): Released /hbase/table-lock/hbase:namespace/write-master:498970000000000 2016-01-11 22:48:27,784 DEBUG [ProcedureExecutor-0] procedure2.ProcedureExecutor(857): Procedure completed in 1.2220sec: CreateTableProcedure (table=hbase:namespace) id=1 owner=elliott state=FINISHED 2016-01-11 22:48:27,785 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=49897] master.MasterRpcServices(1027): Checking to see if procedure is done procId=2 2016-01-11 22:48:27,809 DEBUG [ProcedureExecutor-1] lock.ZKInterProcessLockBase(226): Acquired a lock for /hbase/table-lock/TestSplitWalDataLoss:dataloss/write-master:498970000000000 2016-01-11 22:48:27,814 DEBUG [ProcedureExecutor-1] zookeeper.ZKUtil(627): master:49897-0x1523497b4040000, quorum=localhost:55120, baseZNode=/hbase Unable to get data of znode /hbase/table/TestSplitWalDataLoss:dataloss because node does not exist (not an error) 2016-01-11 22:48:27,888 DEBUG [B.defaultRpcServer.handler=4,queue=0,port=49897] master.MasterRpcServices(1027): Checking to see if procedure is done procId=2 2016-01-11 22:48:28,024 INFO [IPC Server handler 5 on 59295] blockmanagement.BlockManager(2383): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:38785 is added to blk_1073741835_1011{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-940d5c3a-1db1-4402-9e4c-a19c2cc6af02:NORMAL|RBW], ReplicaUnderConstruction[[DISK]DS-2f4161bb-a789-4c3e-bb22-3dbf1babdf25:NORMAL|RBW]]} size 0 2016-01-11 22:48:28,026 INFO [IPC Server handler 8 on 59295] blockmanagement.BlockManager(2383): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:46850 is added to blk_1073741835_1011 size 298 2016-01-11 22:48:28,027 DEBUG [ProcedureExecutor-1] util.FSTableDescriptors(656): Wrote descriptor into: hdfs://localhost:59295/user/elliott/test-data/4f0ddc3b-9bf4-4ec6-b2ab-29e91003a033/.tmp/data/TestSplitWalDataLoss/dataloss/.tabledesc/.tableinfo.0000000001 2016-01-11 22:48:28,029 INFO [RegionOpenAndInitThread-TestSplitWalDataLoss:dataloss-1] regionserver.HRegion(6219): creating HRegion TestSplitWalDataLoss:dataloss HTD == 'TestSplitWalDataLoss:dataloss', {NAME => 'f', BLOOMFILTER => 'ROW', VERSIONS => '1', IN_MEMORY => 'false', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', COMPRESSION => 'NONE', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} RootDir = hdfs://localhost:59295/user/elliott/test-data/4f0ddc3b-9bf4-4ec6-b2ab-29e91003a033/.tmp Table name == TestSplitWalDataLoss:dataloss 2016-01-11 22:48:28,043 INFO [IPC Server handler 8 on 59295] blockmanagement.BlockManager(2383): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:38785 is added to blk_1073741836_1012{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-e3b160d5-0860-46e6-8012-66df73514c0f:NORMAL|RBW], ReplicaUnderConstruction[[DISK]DS-4449a44e-3e96-4db5-b41c-f22c79a44f57:NORMAL|RBW]]} size 0 2016-01-11 22:48:28,044 INFO [IPC Server handler 7 on 59295] blockmanagement.BlockManager(2383): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:46850 is added to blk_1073741836_1012 size 56 2016-01-11 22:48:28,045 DEBUG [RegionOpenAndInitThread-TestSplitWalDataLoss:dataloss-1] regionserver.HRegion(733): Instantiated TestSplitWalDataLoss:dataloss,,1452581307705.bf63661cd1760fafa0bdfe4be2fb3776. 2016-01-11 22:48:28,045 DEBUG [RegionOpenAndInitThread-TestSplitWalDataLoss:dataloss-1] regionserver.HRegion(1389): Closing TestSplitWalDataLoss:dataloss,,1452581307705.bf63661cd1760fafa0bdfe4be2fb3776.: disabling compactions & flushes 2016-01-11 22:48:28,045 DEBUG [RegionOpenAndInitThread-TestSplitWalDataLoss:dataloss-1] regionserver.HRegion(1416): Updates disabled for region TestSplitWalDataLoss:dataloss,,1452581307705.bf63661cd1760fafa0bdfe4be2fb3776. 2016-01-11 22:48:28,045 INFO [RegionOpenAndInitThread-TestSplitWalDataLoss:dataloss-1] regionserver.HRegion(1519): Closed TestSplitWalDataLoss:dataloss,,1452581307705.bf63661cd1760fafa0bdfe4be2fb3776. 2016-01-11 22:48:28,089 DEBUG [B.defaultRpcServer.handler=0,queue=0,port=49897] master.MasterRpcServices(1027): Checking to see if procedure is done procId=2 2016-01-11 22:48:28,213 INFO [ProcedureExecutor-1] hbase.MetaTableAccessor(1199): Added 1 2016-01-11 22:48:28,358 INFO [ProcedureExecutor-1] zookeeper.ZKTableStateManager(100): Moving table TestSplitWalDataLoss:dataloss state from null to ENABLING 2016-01-11 22:48:28,377 DEBUG [ProcedureExecutor-1] balancer.RegionLocationFinder(269): HDFSBlocksDistribution not found in cache for region TestSplitWalDataLoss:dataloss,,1452581307705.bf63661cd1760fafa0bdfe4be2fb3776. 2016-01-11 22:48:28,383 DEBUG [ProcedureExecutor-1] regionserver.HRegionFileSystem(201): No StoreFiles for: hdfs://localhost:59295/user/elliott/test-data/4f0ddc3b-9bf4-4ec6-b2ab-29e91003a033/data/TestSplitWalDataLoss/dataloss/bf63661cd1760fafa0bdfe4be2fb3776/f 2016-01-11 22:48:28,383 INFO [ProcedureExecutor-1] master.AssignmentManager(1613): Assigning 1 region(s) to dev1762.prn2.facebook.com,61792,1452581304511 2016-01-11 22:48:28,383 DEBUG [ProcedureExecutor-1] zookeeper.ZKAssign(174): master:49897-0x1523497b4040000, quorum=localhost:55120, baseZNode=/hbase Async create of unassigned node bf63661cd1760fafa0bdfe4be2fb3776 with OFFLINE state 2016-01-11 22:48:28,390 DEBUG [B.defaultRpcServer.handler=3,queue=0,port=49897] master.MasterRpcServices(1027): Checking to see if procedure is done procId=2 2016-01-11 22:48:28,392 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(554): master:49897-0x1523497b4040000, quorum=localhost:55120, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/region-in-transition 2016-01-11 22:48:28,392 DEBUG [main-EventThread] master.OfflineCallback(69): rs={bf63661cd1760fafa0bdfe4be2fb3776 state=OFFLINE, ts=1452581308376, server=null}, server=dev1762.prn2.facebook.com,61792,1452581304511 2016-01-11 22:48:28,392 DEBUG [main-EventThread] master.OfflineCallback$ExistCallback(106): rs={bf63661cd1760fafa0bdfe4be2fb3776 state=OFFLINE, ts=1452581308376, server=null}, server=dev1762.prn2.facebook.com,61792,1452581304511 2016-01-11 22:48:28,393 DEBUG [ProcedureExecutor-1] master.AssignmentManager(1666): dev1762.prn2.facebook.com,61792,1452581304511 unassigned znodes=1 of total=1; oldCounter=0 2016-01-11 22:48:28,394 INFO [ProcedureExecutor-1] master.RegionStates(1109): Transition {bf63661cd1760fafa0bdfe4be2fb3776 state=OFFLINE, ts=1452581308383, server=null} to {bf63661cd1760fafa0bdfe4be2fb3776 state=PENDING_OPEN, ts=1452581308394, server=dev1762.prn2.facebook.com,61792,1452581304511} 2016-01-11 22:48:28,395 INFO [PriorityRpcServer.handler=3,queue=1,port=61792] regionserver.RSRpcServices(1557): Open TestSplitWalDataLoss:dataloss,,1452581307705.bf63661cd1760fafa0bdfe4be2fb3776. 2016-01-11 22:48:28,399 DEBUG [RS_OPEN_REGION-dev1762:61792-1] zookeeper.ZKAssign(805): regionserver:61792-0x1523497b4040001, quorum=localhost:55120, baseZNode=/hbase Transitioning bf63661cd1760fafa0bdfe4be2fb3776 from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING 2016-01-11 22:48:28,408 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(554): master:49897-0x1523497b4040000, quorum=localhost:55120, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/region-in-transition/bf63661cd1760fafa0bdfe4be2fb3776 2016-01-11 22:48:28,408 DEBUG [RS_OPEN_REGION-dev1762:61792-1] zookeeper.ZKAssign(880): regionserver:61792-0x1523497b4040001, quorum=localhost:55120, baseZNode=/hbase Transitioned node bf63661cd1760fafa0bdfe4be2fb3776 from M_ZK_REGION_OFFLINE to RS_ZK_REGION_OPENING 2016-01-11 22:48:28,409 DEBUG [RS_OPEN_REGION-dev1762:61792-1] regionserver.HRegion(6408): Opening region: {ENCODED => bf63661cd1760fafa0bdfe4be2fb3776, NAME => 'TestSplitWalDataLoss:dataloss,,1452581307705.bf63661cd1760fafa0bdfe4be2fb3776.', STARTKEY => '', ENDKEY => ''} 2016-01-11 22:48:28,409 DEBUG [AM.ZK.Worker-pool2-t10] master.AssignmentManager(956): Handling RS_ZK_REGION_OPENING, server=dev1762.prn2.facebook.com,61792,1452581304511, region=bf63661cd1760fafa0bdfe4be2fb3776, current_state={bf63661cd1760fafa0bdfe4be2fb3776 state=PENDING_OPEN, ts=1452581308394, server=dev1762.prn2.facebook.com,61792,1452581304511} 2016-01-11 22:48:28,409 INFO [AM.ZK.Worker-pool2-t10] master.RegionStates(1109): Transition {bf63661cd1760fafa0bdfe4be2fb3776 state=PENDING_OPEN, ts=1452581308394, server=dev1762.prn2.facebook.com,61792,1452581304511} to {bf63661cd1760fafa0bdfe4be2fb3776 state=OPENING, ts=1452581308409, server=dev1762.prn2.facebook.com,61792,1452581304511} 2016-01-11 22:48:28,409 DEBUG [RS_OPEN_REGION-dev1762:61792-1] regionserver.MetricsRegionSourceImpl(70): Creating new MetricsRegionSourceImpl for table dataloss bf63661cd1760fafa0bdfe4be2fb3776 2016-01-11 22:48:28,409 DEBUG [RS_OPEN_REGION-dev1762:61792-1] regionserver.HRegion(733): Instantiated TestSplitWalDataLoss:dataloss,,1452581307705.bf63661cd1760fafa0bdfe4be2fb3776. 2016-01-11 22:48:28,413 INFO [StoreOpener-bf63661cd1760fafa0bdfe4be2fb3776-1] hfile.CacheConfig(292): blockCache=LruBlockCache{blockCount=0, currentSize=1071840, freeSize=1042890464, maxSize=1043962304, heapSize=1071840, minSize=991764160, minFactor=0.95, multiSize=495882080, multiFactor=0.5, singleSize=247941040, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false 2016-01-11 22:48:28,413 INFO [StoreOpener-bf63661cd1760fafa0bdfe4be2fb3776-1] compactions.CompactionConfiguration(104): size [134217728, 9223372036854775807, 9223372036854775807); files [3, 10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000 2016-01-11 22:48:28,415 DEBUG [StoreOpener-bf63661cd1760fafa0bdfe4be2fb3776-1] regionserver.HRegionFileSystem(201): No StoreFiles for: hdfs://localhost:59295/user/elliott/test-data/4f0ddc3b-9bf4-4ec6-b2ab-29e91003a033/data/TestSplitWalDataLoss/dataloss/bf63661cd1760fafa0bdfe4be2fb3776/f 2016-01-11 22:48:28,416 DEBUG [RS_OPEN_REGION-dev1762:61792-1] regionserver.HRegion(3890): Found 0 recovered edits file(s) under hdfs://localhost:59295/user/elliott/test-data/4f0ddc3b-9bf4-4ec6-b2ab-29e91003a033/data/TestSplitWalDataLoss/dataloss/bf63661cd1760fafa0bdfe4be2fb3776 2016-01-11 22:48:28,418 DEBUG [RS_OPEN_REGION-dev1762:61792-1] regionserver.FlushLargeStoresPolicy(56): hbase.hregion.percolumnfamilyflush.size.lower.bound is not specified, use global config(16777216) instead 2016-01-11 22:48:28,422 DEBUG [RS_OPEN_REGION-dev1762:61792-1] wal.WALSplitter(729): Wrote region seqId=hdfs://localhost:59295/user/elliott/test-data/4f0ddc3b-9bf4-4ec6-b2ab-29e91003a033/data/TestSplitWalDataLoss/dataloss/bf63661cd1760fafa0bdfe4be2fb3776/recovered.edits/2.seqid to file, newSeqId=2, maxSeqId=0 2016-01-11 22:48:28,422 INFO [RS_OPEN_REGION-dev1762:61792-1] regionserver.HRegion(868): Onlined bf63661cd1760fafa0bdfe4be2fb3776; next sequenceid=2 2016-01-11 22:48:28,424 DEBUG [RS_OPEN_REGION-dev1762:61792-1] zookeeper.ZKAssign(643): regionserver:61792-0x1523497b4040001, quorum=localhost:55120, baseZNode=/hbase Attempting to retransition opening state of node bf63661cd1760fafa0bdfe4be2fb3776 2016-01-11 22:48:28,424 INFO [PostOpenDeployTasks:bf63661cd1760fafa0bdfe4be2fb3776] regionserver.HRegionServer(1912): Post open deploy tasks for TestSplitWalDataLoss:dataloss,,1452581307705.bf63661cd1760fafa0bdfe4be2fb3776. 2016-01-11 22:48:28,427 INFO [PostOpenDeployTasks:bf63661cd1760fafa0bdfe4be2fb3776] hbase.MetaTableAccessor(1394): Updated row TestSplitWalDataLoss:dataloss,,1452581307705.bf63661cd1760fafa0bdfe4be2fb3776. with server=dev1762.prn2.facebook.com,61792,1452581304511 2016-01-11 22:48:28,427 DEBUG [PostOpenDeployTasks:bf63661cd1760fafa0bdfe4be2fb3776] regionserver.HRegionServer(1946): Finished post open deploy task for TestSplitWalDataLoss:dataloss,,1452581307705.bf63661cd1760fafa0bdfe4be2fb3776. 2016-01-11 22:48:28,427 DEBUG [RS_OPEN_REGION-dev1762:61792-1] zookeeper.ZKAssign(805): regionserver:61792-0x1523497b4040001, quorum=localhost:55120, baseZNode=/hbase Transitioning bf63661cd1760fafa0bdfe4be2fb3776 from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENED 2016-01-11 22:48:28,433 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(554): master:49897-0x1523497b4040000, quorum=localhost:55120, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/region-in-transition/bf63661cd1760fafa0bdfe4be2fb3776 2016-01-11 22:48:28,433 DEBUG [RS_OPEN_REGION-dev1762:61792-1] zookeeper.ZKAssign(880): regionserver:61792-0x1523497b4040001, quorum=localhost:55120, baseZNode=/hbase Transitioned node bf63661cd1760fafa0bdfe4be2fb3776 from RS_ZK_REGION_OPENING to RS_ZK_REGION_OPENED 2016-01-11 22:48:28,433 DEBUG [RS_OPEN_REGION-dev1762:61792-1] coordination.ZkOpenRegionCoordination(91): Transitioned bf63661cd1760fafa0bdfe4be2fb3776 to OPENED in zk on dev1762.prn2.facebook.com,61792,1452581304511 2016-01-11 22:48:28,434 DEBUG [RS_OPEN_REGION-dev1762:61792-1] handler.OpenRegionHandler(170): Opened TestSplitWalDataLoss:dataloss,,1452581307705.bf63661cd1760fafa0bdfe4be2fb3776. on dev1762.prn2.facebook.com,61792,1452581304511 2016-01-11 22:48:28,434 DEBUG [AM.ZK.Worker-pool2-t11] master.AssignmentManager(956): Handling RS_ZK_REGION_OPENED, server=dev1762.prn2.facebook.com,61792,1452581304511, region=bf63661cd1760fafa0bdfe4be2fb3776, current_state={bf63661cd1760fafa0bdfe4be2fb3776 state=OPENING, ts=1452581308409, server=dev1762.prn2.facebook.com,61792,1452581304511} 2016-01-11 22:48:28,434 INFO [AM.ZK.Worker-pool2-t11] master.RegionStates(1109): Transition {bf63661cd1760fafa0bdfe4be2fb3776 state=OPENING, ts=1452581308409, server=dev1762.prn2.facebook.com,61792,1452581304511} to {bf63661cd1760fafa0bdfe4be2fb3776 state=OPEN, ts=1452581308434, server=dev1762.prn2.facebook.com,61792,1452581304511} 2016-01-11 22:48:28,434 DEBUG [AM.ZK.Worker-pool2-t11] coordination.ZkOpenRegionCoordination(350): Handling OPENED of bf63661cd1760fafa0bdfe4be2fb3776 from dev1762.prn2.facebook.com,49897,1452581304150; deleting unassigned node 2016-01-11 22:48:28,442 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(554): master:49897-0x1523497b4040000, quorum=localhost:55120, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/region-in-transition/bf63661cd1760fafa0bdfe4be2fb3776 2016-01-11 22:48:28,442 DEBUG [AM.ZK.Worker-pool2-t11] zookeeper.ZKAssign(479): master:49897-0x1523497b4040000, quorum=localhost:55120, baseZNode=/hbase Deleted unassigned node bf63661cd1760fafa0bdfe4be2fb3776 in expected state RS_ZK_REGION_OPENED 2016-01-11 22:48:28,442 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(554): master:49897-0x1523497b4040000, quorum=localhost:55120, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/region-in-transition 2016-01-11 22:48:28,442 DEBUG [AM.ZK.Worker-pool2-t12] master.AssignmentManager$4(1345): Znode TestSplitWalDataLoss:dataloss,,1452581307705.bf63661cd1760fafa0bdfe4be2fb3776. deleted, state: {bf63661cd1760fafa0bdfe4be2fb3776 state=OPEN, ts=1452581308434, server=dev1762.prn2.facebook.com,61792,1452581304511} 2016-01-11 22:48:28,442 DEBUG [AM.ZK.Worker-pool2-t12] master.RegionStates(456): Onlined bf63661cd1760fafa0bdfe4be2fb3776 on dev1762.prn2.facebook.com,61792,1452581304511 2016-01-11 22:48:28,442 DEBUG [ProcedureExecutor-1] master.AssignmentManager(1808): Bulk assigning done for dev1762.prn2.facebook.com,61792,1452581304511 2016-01-11 22:48:28,442 INFO [ProcedureExecutor-1] zookeeper.ZKTableStateManager(100): Moving table TestSplitWalDataLoss:dataloss state from ENABLING to ENABLED 2016-01-11 22:48:28,892 DEBUG [ProcedureExecutor-1] lock.ZKInterProcessLockBase(328): Released /hbase/table-lock/TestSplitWalDataLoss:dataloss/write-master:498970000000000 2016-01-11 22:48:28,892 DEBUG [B.defaultRpcServer.handler=1,queue=0,port=49897] master.MasterRpcServices(1027): Checking to see if procedure is done procId=2 2016-01-11 22:48:28,892 DEBUG [ProcedureExecutor-1] procedure2.ProcedureExecutor(857): Procedure completed in 1.0340sec: CreateTableProcedure (table=TestSplitWalDataLoss:dataloss) id=2 owner=elliott state=FINISHED 2016-01-11 22:48:29,893 DEBUG [B.defaultRpcServer.handler=2,queue=0,port=49897] master.MasterRpcServices(1027): Checking to see if procedure is done procId=2 2016-01-11 22:48:29,894 INFO [main] client.HBaseAdmin$CreateTableFuture(770): Created TestSplitWalDataLoss:dataloss 2016-01-11 22:48:29,897 DEBUG [htable-pool13-t1] ipc.RpcClientImpl$Connection(351): Use SIMPLE authentication for service ClientService, sasl=false 2016-01-11 22:48:29,898 DEBUG [htable-pool13-t1] ipc.RpcClientImpl$Connection(716): Connecting to dev1762.prn2.facebook.com/10.35.11.29:61792 2016-01-11 22:48:29,898 DEBUG [RpcServer.listener,port=61792] ipc.RpcServer$Listener(879): RpcServer.listener,port=61792: connection from 10.35.11.29:60626; # active connections: 3 2016-01-11 22:48:29,899 INFO [RpcServer.reader=1,bindAddress=dev1762.prn2.facebook.com,port=61792] ipc.RpcServer$Connection(1712): Auth successful for elliott (auth:SIMPLE) 2016-01-11 22:48:29,900 INFO [RpcServer.reader=1,bindAddress=dev1762.prn2.facebook.com,port=61792] ipc.RpcServer$Connection(1742): Connection from 10.35.11.29 port: 60626 with version info: version: "1.2.0" url: "git://dev1762.prn2.facebook.com/data/users/elliott/hbase" revision: "4b59f62c231a39fd4a824c0cbf4c2166c4ec2a76" user: "elliott" date: "Mon Jan 11 22:47:43 PST 2016" src_checksum: "58511d3ee667dd7df285c172560770ca" 2016-01-11 22:48:29,905 INFO [main] zookeeper.RecoverableZooKeeper(120): Process identifier=hconnection-0x77bd7fe7 connecting to ZooKeeper ensemble=localhost:55120 2016-01-11 22:48:29,925 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(554): hconnection-0x77bd7fe70x0, quorum=localhost:55120, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2016-01-11 22:48:29,926 DEBUG [main] ipc.AbstractRpcClient(115): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@35d3ab60, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2016-01-11 22:48:29,927 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(638): hconnection-0x77bd7fe7-0x1523497b404000a connected 2016-01-11 22:48:29,928 DEBUG [htable-pool14-t1] ipc.RpcClientImpl$Connection(351): Use SIMPLE authentication for service ClientService, sasl=false 2016-01-11 22:48:29,928 DEBUG [htable-pool14-t1] ipc.RpcClientImpl$Connection(716): Connecting to dev1762.prn2.facebook.com/10.35.11.29:61792 2016-01-11 22:48:29,928 DEBUG [RpcServer.listener,port=61792] ipc.RpcServer$Listener(879): RpcServer.listener,port=61792: connection from 10.35.11.29:60628; # active connections: 4 2016-01-11 22:48:29,929 INFO [RpcServer.reader=2,bindAddress=dev1762.prn2.facebook.com,port=61792] ipc.RpcServer$Connection(1712): Auth successful for elliott (auth:SIMPLE) 2016-01-11 22:48:29,929 INFO [RpcServer.reader=2,bindAddress=dev1762.prn2.facebook.com,port=61792] ipc.RpcServer$Connection(1742): Connection from 10.35.11.29 port: 60628 with version info: version: "1.2.0" url: "git://dev1762.prn2.facebook.com/data/users/elliott/hbase" revision: "4b59f62c231a39fd4a824c0cbf4c2166c4ec2a76" user: "elliott" date: "Mon Jan 11 22:47:43 PST 2016" src_checksum: "58511d3ee667dd7df285c172560770ca" 2016-01-11 22:48:29,933 INFO [main] hbase.HBaseTestingUtility(2529): getMetaTableRows: row -> TestSplitWalDataLoss:dataloss,,1452581307705.bf63661cd1760fafa0bdfe4be2fb3776.{ENCODED => bf63661cd1760fafa0bdfe4be2fb3776, NAME => 'TestSplitWalDataLoss:dataloss,,1452581307705.bf63661cd1760fafa0bdfe4be2fb3776.', STARTKEY => '', ENDKEY => ''} 2016-01-11 22:48:29,934 INFO [main] client.ConnectionManager$HConnectionImplementation(1712): Closing zookeeper sessionid=0x1523497b404000a 2016-01-11 22:48:29,942 DEBUG [main] ipc.RpcClientImpl(1147): Stopping rpc client 2016-01-11 22:48:29,942 DEBUG [RpcServer.reader=2,bindAddress=dev1762.prn2.facebook.com,port=61792] ipc.RpcServer$Listener(911): RpcServer.listener,port=61792: DISCONNECTING client 10.35.11.29:60628 because read count=-1. Number of active connections: 4 2016-01-11 22:48:29,952 DEBUG [main] hbase.HBaseTestingUtility(2556): Found 1 rows for table TestSplitWalDataLoss:dataloss 2016-01-11 22:48:29,952 DEBUG [main] hbase.HBaseTestingUtility(2559): FirstRow=TestSplitWalDataLoss:dataloss,,1452581307705.bf63661cd1760fafa0bdfe4be2fb3776. 2016-01-11 22:48:30,190 DEBUG [main] client.ClientSmallReversedScanner(147): Finished with small scan at {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''} 2016-01-11 22:48:30,191 ERROR [main] client.AsyncProcess(404): Failed to get region location org.apache.hadoop.hbase.TableNotFoundException: TestSplitWalDataLoss:dataloss at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegionInMeta(ConnectionManager.java:1286) at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1184) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:394) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:343) at org.apache.hadoop.hbase.client.BufferedMutatorImpl.backgroundFlushCommits(BufferedMutatorImpl.java:238) at org.apache.hadoop.hbase.client.BufferedMutatorImpl.flush(BufferedMutatorImpl.java:190) at org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:1430) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:1021) at org.apache.hadoop.hbase.regionserver.TestSplitWalDataLoss.test(TestSplitWalDataLoss.java:124) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:497) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.runners.ParentRunner.run(ParentRunner.java:363) at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:283) at org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:173) at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:153) at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:128) at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:203) at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:155) at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:103) 2016-01-11 22:48:30,196 INFO [main] hbase.HBaseTestingUtility(1090): Shutting down minicluster 2016-01-11 22:48:30,196 INFO [main] client.ConnectionManager$HConnectionImplementation(2141): Closing master protocol: MasterService 2016-01-11 22:48:30,196 INFO [main] client.ConnectionManager$HConnectionImplementation(1712): Closing zookeeper sessionid=0x1523497b4040009 2016-01-11 22:48:30,200 DEBUG [main] ipc.RpcClientImpl(1147): Stopping rpc client 2016-01-11 22:48:30,201 DEBUG [RpcServer.reader=1,bindAddress=dev1762.prn2.facebook.com,port=61792] ipc.RpcServer$Listener(911): RpcServer.listener,port=61792: DISCONNECTING client 10.35.11.29:60626 because read count=-1. Number of active connections: 3 2016-01-11 22:48:30,201 DEBUG [RpcServer.reader=0,bindAddress=dev1762.prn2.facebook.com,port=49897] ipc.RpcServer$Listener(911): RpcServer.listener,port=49897: DISCONNECTING client 10.35.11.29:47737 because read count=-1. Number of active connections: 3 2016-01-11 22:48:30,211 DEBUG [main] util.JVMClusterUtil(241): Shutting down HBase Cluster 2016-01-11 22:48:30,211 INFO [main] regionserver.HRegionServer(1878): STOPPED: Cluster shutdown requested 2016-01-11 22:48:30,212 INFO [M:0;dev1762:49897] procedure2.ProcedureExecutor(517): Stopping the procedure executor 2016-01-11 22:48:30,212 INFO [M:0;dev1762:49897] wal.WALProcedureStore(192): Stopping the WAL Procedure Store 2016-01-11 22:48:30,217 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(554): master:49897-0x1523497b4040000, quorum=localhost:55120, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2016-01-11 22:48:30,217 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(554): regionserver:33025-0x1523497b4040002, quorum=localhost:55120, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2016-01-11 22:48:30,217 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(554): regionserver:61792-0x1523497b4040001, quorum=localhost:55120, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2016-01-11 22:48:30,217 INFO [main] regionserver.HRegionServer(1878): STOPPED: Shutdown requested 2016-01-11 22:48:30,217 DEBUG [main-EventThread] zookeeper.ZKUtil(370): master:49897-0x1523497b4040000, quorum=localhost:55120, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2016-01-11 22:48:30,217 DEBUG [main-EventThread] zookeeper.ZKUtil(370): regionserver:61792-0x1523497b4040001, quorum=localhost:55120, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2016-01-11 22:48:30,217 INFO [RS:0;dev1762:61792] regionserver.SplitLogWorker(164): Sending interrupt to stop the worker thread 2016-01-11 22:48:30,217 INFO [main] regionserver.HRegionServer(1878): STOPPED: Shutdown requested 2016-01-11 22:48:30,217 DEBUG [main-EventThread] zookeeper.ZKUtil(370): regionserver:33025-0x1523497b4040002, quorum=localhost:55120, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2016-01-11 22:48:30,218 INFO [RS:1;dev1762:33025] regionserver.SplitLogWorker(164): Sending interrupt to stop the worker thread 2016-01-11 22:48:30,218 INFO [SplitLogWorker-dev1762:61792] regionserver.SplitLogWorker(146): SplitLogWorker interrupted. Exiting. 2016-01-11 22:48:30,218 INFO [SplitLogWorker-dev1762:33025] regionserver.SplitLogWorker(146): SplitLogWorker interrupted. Exiting. 2016-01-11 22:48:30,218 INFO [RS:1;dev1762:33025] regionserver.HeapMemoryManager(202): Stoping HeapMemoryTuner chore. 2016-01-11 22:48:30,218 INFO [RS:0;dev1762:61792] regionserver.HeapMemoryManager(202): Stoping HeapMemoryTuner chore. 2016-01-11 22:48:30,218 INFO [SplitLogWorker-dev1762:33025] regionserver.SplitLogWorker(155): SplitLogWorker dev1762.prn2.facebook.com,33025,1452581304563 exiting 2016-01-11 22:48:30,218 INFO [SplitLogWorker-dev1762:61792] regionserver.SplitLogWorker(155): SplitLogWorker dev1762.prn2.facebook.com,61792,1452581304511 exiting 2016-01-11 22:48:30,219 INFO [RS:1;dev1762:33025] flush.RegionServerFlushTableProcedureManager(115): Stopping region server flush procedure manager gracefully. 2016-01-11 22:48:30,219 INFO [MemStoreFlusher.1] regionserver.MemStoreFlusher$FlushHandler(280): MemStoreFlusher.1 exiting 2016-01-11 22:48:30,219 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(280): MemStoreFlusher.0 exiting 2016-01-11 22:48:30,219 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(280): MemStoreFlusher.0 exiting 2016-01-11 22:48:30,219 INFO [MemStoreFlusher.1] regionserver.MemStoreFlusher$FlushHandler(280): MemStoreFlusher.1 exiting 2016-01-11 22:48:30,219 INFO [RS:0;dev1762:61792] flush.RegionServerFlushTableProcedureManager(115): Stopping region server flush procedure manager gracefully. 2016-01-11 22:48:30,219 INFO [RS:1;dev1762:33025] snapshot.RegionServerSnapshotManager(136): Stopping RegionServerSnapshotManager gracefully. 2016-01-11 22:48:30,220 INFO [RS:0;dev1762:61792] snapshot.RegionServerSnapshotManager(136): Stopping RegionServerSnapshotManager gracefully. 2016-01-11 22:48:30,220 INFO [RS:1;dev1762:33025] regionserver.HRegionServer(1032): stopping server dev1762.prn2.facebook.com,33025,1452581304563 2016-01-11 22:48:30,220 DEBUG [RS:1;dev1762:33025] zookeeper.MetaTableLocator(616): Stopping MetaTableLocator 2016-01-11 22:48:30,220 INFO [RS:1;dev1762:33025] client.ConnectionManager$HConnectionImplementation(1712): Closing zookeeper sessionid=0x1523497b4040004 2016-01-11 22:48:30,223 INFO [RS:0;dev1762:61792] regionserver.HRegionServer(2809): Received CLOSE for the region: bf63661cd1760fafa0bdfe4be2fb3776, which we are already trying to CLOSE, but not completed yet 2016-01-11 22:48:30,223 WARN [RS:0;dev1762:61792] regionserver.HRegionServer(2748): Failed to close TestSplitWalDataLoss:dataloss,,1452581307705.bf63661cd1760fafa0bdfe4be2fb3776. - ignoring and continuing org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: The region bf63661cd1760fafa0bdfe4be2fb3776 was already closing. New CLOSE request is ignored. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2817) at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegionIgnoreErrors(HRegionServer.java:2743) at org.apache.hadoop.hbase.regionserver.HRegionServer.closeUserRegions(HRegionServer.java:2375) at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:1031) at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.runRegionServer(MiniHBaseCluster.java:156) at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.access$000(MiniHBaseCluster.java:108) at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer$1.run(MiniHBaseCluster.java:140) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:360) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1594) at org.apache.hadoop.hbase.security.User$SecureHadoopUser.runAs(User.java:334) at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.run(MiniHBaseCluster.java:138) at java.lang.Thread.run(Thread.java:745) 2016-01-11 22:48:30,223 DEBUG [RS_CLOSE_REGION-dev1762:61792-0] handler.CloseRegionHandler(119): Processing close of TestSplitWalDataLoss:dataloss,,1452581307705.bf63661cd1760fafa0bdfe4be2fb3776. 2016-01-11 22:48:30,224 INFO [RS:0;dev1762:61792] regionserver.HRegionServer(1032): stopping server dev1762.prn2.facebook.com,61792,1452581304511 2016-01-11 22:48:30,224 DEBUG [RS_CLOSE_REGION-dev1762:61792-1] handler.CloseRegionHandler(119): Processing close of hbase:namespace,,1452581306381.8fdaad26859458841be03061163dcfe7. 2016-01-11 22:48:30,224 DEBUG [RS:0;dev1762:61792] zookeeper.MetaTableLocator(616): Stopping MetaTableLocator 2016-01-11 22:48:30,225 DEBUG [RS_CLOSE_REGION-dev1762:61792-0] regionserver.HRegion(1389): Closing TestSplitWalDataLoss:dataloss,,1452581307705.bf63661cd1760fafa0bdfe4be2fb3776.: disabling compactions & flushes 2016-01-11 22:48:30,225 INFO [RS:0;dev1762:61792] client.ConnectionManager$HConnectionImplementation(1712): Closing zookeeper sessionid=0x1523497b4040005 2016-01-11 22:48:30,226 DEBUG [RS:1;dev1762:33025] ipc.RpcClientImpl(1147): Stopping rpc client 2016-01-11 22:48:30,226 DEBUG [RS_CLOSE_REGION-dev1762:61792-0] regionserver.HRegion(1416): Updates disabled for region TestSplitWalDataLoss:dataloss,,1452581307705.bf63661cd1760fafa0bdfe4be2fb3776. 2016-01-11 22:48:30,226 DEBUG [RS_CLOSE_REGION-dev1762:61792-1] regionserver.HRegion(1389): Closing hbase:namespace,,1452581306381.8fdaad26859458841be03061163dcfe7.: disabling compactions & flushes 2016-01-11 22:48:30,226 INFO [RS:1;dev1762:33025] regionserver.HRegionServer(1060): stopping server dev1762.prn2.facebook.com,33025,1452581304563; all regions closed. 2016-01-11 22:48:30,227 DEBUG [RS_CLOSE_REGION-dev1762:61792-1] regionserver.HRegion(1416): Updates disabled for region hbase:namespace,,1452581306381.8fdaad26859458841be03061163dcfe7. 2016-01-11 22:48:30,228 INFO [StoreCloserThread-TestSplitWalDataLoss:dataloss,,1452581307705.bf63661cd1760fafa0bdfe4be2fb3776.-1] regionserver.HStore(875): Closed f 2016-01-11 22:48:30,228 DEBUG [RS:1;dev1762:33025] wal.FSHLog(1063): Closing WAL writer in /user/elliott/test-data/4f0ddc3b-9bf4-4ec6-b2ab-29e91003a033/WALs/dev1762.prn2.facebook.com,33025,1452581304563 2016-01-11 22:48:30,228 INFO [RS_CLOSE_REGION-dev1762:61792-1] regionserver.HRegion(2157): Flushing 1/1 column families, memstore=544 B 2016-01-11 22:48:30,229 INFO [IPC Server handler 0 on 59295] blockmanagement.BlockManager(2383): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:38785 is added to blk_1073741832_1008{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-4449a44e-3e96-4db5-b41c-f22c79a44f57:NORMAL|RBW], ReplicaUnderConstruction[[DISK]DS-e3b160d5-0860-46e6-8012-66df73514c0f:NORMAL|RBW]]} size 473 2016-01-11 22:48:30,231 INFO [IPC Server handler 3 on 59295] blockmanagement.BlockManager(2383): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:46850 is added to blk_1073741832_1008 size 6871 2016-01-11 22:48:30,232 INFO [M:0;dev1762:49897] regionserver.HRegionServer(1032): stopping server dev1762.prn2.facebook.com,49897,1452581304150 2016-01-11 22:48:30,232 DEBUG [M:0;dev1762:49897] zookeeper.MetaTableLocator(616): Stopping MetaTableLocator 2016-01-11 22:48:30,235 DEBUG [RS:0;dev1762:61792] ipc.RpcClientImpl(1147): Stopping rpc client 2016-01-11 22:48:30,236 INFO [IPC Server handler 4 on 59295] blockmanagement.BlockManager(2383): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:46850 is added to blk_1073741830_1006{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-e3b160d5-0860-46e6-8012-66df73514c0f:NORMAL|RBW], ReplicaUnderConstruction[[DISK]DS-4449a44e-3e96-4db5-b41c-f22c79a44f57:NORMAL|RBW]]} size 83 2016-01-11 22:48:30,237 INFO [RS:0;dev1762:61792] regionserver.CompactSplitThread(403): Waiting for Split Thread to finish... 2016-01-11 22:48:30,236 INFO [M:0;dev1762:49897] client.ConnectionManager$HConnectionImplementation(1712): Closing zookeeper sessionid=0x1523497b4040003 2016-01-11 22:48:30,237 INFO [RS:0;dev1762:61792] regionserver.CompactSplitThread(403): Waiting for Merge Thread to finish... 2016-01-11 22:48:30,238 INFO [RS:0;dev1762:61792] regionserver.CompactSplitThread(403): Waiting for Large Compaction Thread to finish... 2016-01-11 22:48:30,238 INFO [RS:0;dev1762:61792] regionserver.CompactSplitThread(403): Waiting for Small Compaction Thread to finish... 2016-01-11 22:48:30,238 INFO [IPC Server handler 9 on 59295] blockmanagement.BlockManager(2383): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:38785 is added to blk_1073741830_1006 size 91 2016-01-11 22:48:30,238 INFO [RS:0;dev1762:61792] regionserver.HRegionServer(1261): Waiting on 3 regions to close 2016-01-11 22:48:30,238 DEBUG [RS:0;dev1762:61792] regionserver.HRegionServer(1265): {1588230740=TestSplitWalDataLoss:dataloss,,1452581307705.bf63661cd1760fafa0bdfe4be2fb3776., bf63661cd1760fafa0bdfe4be2fb3776=TestSplitWalDataLoss:dataloss,,1452581307705.bf63661cd1760fafa0bdfe4be2fb3776., 8fdaad26859458841be03061163dcfe7=hbase:namespace,,1452581306381.8fdaad26859458841be03061163dcfe7.} 2016-01-11 22:48:30,241 DEBUG [M:0;dev1762:49897] ipc.RpcClientImpl(1147): Stopping rpc client 2016-01-11 22:48:30,242 DEBUG [RpcServer.reader=2,bindAddress=dev1762.prn2.facebook.com,port=61792] ipc.RpcServer$Listener(911): RpcServer.listener,port=61792: DISCONNECTING client 10.35.11.29:60591 because read count=-1. Number of active connections: 2 2016-01-11 22:48:30,242 DEBUG [RS_CLOSE_REGION-dev1762:61792-0] wal.WALSplitter(729): Wrote region seqId=hdfs://localhost:59295/user/elliott/test-data/4f0ddc3b-9bf4-4ec6-b2ab-29e91003a033/data/TestSplitWalDataLoss/dataloss/bf63661cd1760fafa0bdfe4be2fb3776/recovered.edits/4.seqid to file, newSeqId=4, maxSeqId=2 2016-01-11 22:48:30,245 DEBUG [RS:1;dev1762:33025] wal.FSHLog(1021): Moved 1 WAL file(s) to /user/elliott/test-data/4f0ddc3b-9bf4-4ec6-b2ab-29e91003a033/oldWALs 2016-01-11 22:48:30,245 INFO [RS:1;dev1762:33025] wal.FSHLog(1024): Closed WAL: FSHLog dev1762.prn2.facebook.com%2C33025%2C1452581304563.default:(num 1452581305237) 2016-01-11 22:48:30,245 DEBUG [RS:1;dev1762:33025] ipc.RpcClientImpl(1147): Stopping rpc client 2016-01-11 22:48:30,246 DEBUG [RpcServer.reader=2,bindAddress=dev1762.prn2.facebook.com,port=49897] ipc.RpcServer$Listener(911): RpcServer.listener,port=49897: DISCONNECTING client 10.35.11.29:56116 because read count=-1. Number of active connections: 2 2016-01-11 22:48:30,246 INFO [RS_CLOSE_REGION-dev1762:61792-0] regionserver.HRegion(1519): Closed TestSplitWalDataLoss:dataloss,,1452581307705.bf63661cd1760fafa0bdfe4be2fb3776. 2016-01-11 22:48:30,246 DEBUG [RS_CLOSE_REGION-dev1762:61792-0] handler.CloseRegionHandler(164): Closed TestSplitWalDataLoss:dataloss,,1452581307705.bf63661cd1760fafa0bdfe4be2fb3776. 2016-01-11 22:48:30,252 INFO [M:0;dev1762:49897] regionserver.HRegionServer(1060): stopping server dev1762.prn2.facebook.com,49897,1452581304150; all regions closed. 2016-01-11 22:48:30,253 INFO [M:0;dev1762:49897] hbase.ChoreService(323): Chore service for: dev1762.prn2.facebook.com,49897,1452581304150 had [[ScheduledChore: Name: dev1762.prn2.facebook.com,49897,1452581304150-RegionNormalizerChore Period: 1800000 Unit: MILLISECONDS], [ScheduledChore: Name: LogsCleaner Period: 60000 Unit: MILLISECONDS], [ScheduledChore: Name: HFileCleaner Period: 60000 Unit: MILLISECONDS], [ScheduledChore: Name: dev1762.prn2.facebook.com,49897,1452581304150-BalancerChore Period: 300000 Unit: MILLISECONDS], [ScheduledChore: Name: dev1762.prn2.facebook.com,49897,1452581304150-ClusterStatusChore Period: 60000 Unit: MILLISECONDS], [ScheduledChore: Name: CatalogJanitor-dev1762:49897 Period: 300000 Unit: MILLISECONDS]] on shutdown 2016-01-11 22:48:30,253 INFO [M:0;dev1762:49897] master.ServerManager(557): Waiting on regionserver(s) to go down dev1762.prn2.facebook.com,33025,1452581304563, dev1762.prn2.facebook.com,61792,1452581304511 2016-01-11 22:48:30,255 INFO [RS:1;dev1762:33025] regionserver.Leases(146): RS:1;dev1762:33025 closing leases 2016-01-11 22:48:30,256 INFO [RS:1;dev1762:33025] regionserver.Leases(149): RS:1;dev1762:33025 closed leases 2016-01-11 22:48:30,256 INFO [RS:1;dev1762:33025] hbase.ChoreService(323): Chore service for: dev1762.prn2.facebook.com,33025,1452581304563 had [[ScheduledChore: Name: dev1762.prn2.facebook.com,33025,1452581304563-MemstoreFlusherChore Period: 1000 Unit: MILLISECONDS], [ScheduledChore: Name: MovedRegionsCleaner for region dev1762.prn2.facebook.com,33025,1452581304563 Period: 120000 Unit: MILLISECONDS]] on shutdown 2016-01-11 22:48:30,288 INFO [IPC Server handler 3 on 59295] blockmanagement.BlockManager(2383): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:38785 is added to blk_1073741837_1013{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-2f4161bb-a789-4c3e-bb22-3dbf1babdf25:NORMAL|RBW], ReplicaUnderConstruction[[DISK]DS-940d5c3a-1db1-4402-9e4c-a19c2cc6af02:NORMAL|RBW]]} size 0 2016-01-11 22:48:30,288 INFO [IPC Server handler 5 on 59295] blockmanagement.BlockManager(2383): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:46850 is added to blk_1073741837_1013{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-2f4161bb-a789-4c3e-bb22-3dbf1babdf25:NORMAL|RBW], ReplicaUnderConstruction[[DISK]DS-940d5c3a-1db1-4402-9e4c-a19c2cc6af02:NORMAL|RBW]]} size 0 2016-01-11 22:48:30,289 INFO [RS_CLOSE_REGION-dev1762:61792-1] regionserver.DefaultStoreFlusher(90): Flushed, sequenceid=8, memsize=544, hasBloomFilter=true, into tmp file hdfs://localhost:59295/user/elliott/test-data/4f0ddc3b-9bf4-4ec6-b2ab-29e91003a033/data/hbase/namespace/8fdaad26859458841be03061163dcfe7/.tmp/00c3373677094307a4be61e0469ea596 2016-01-11 22:48:30,309 INFO [regionserver/dev1762.prn2.facebook.com/10.35.11.29:0.logRoller] regionserver.LogRoller(176): LogRoller exiting. 2016-01-11 22:48:30,309 INFO [regionserver/dev1762.prn2.facebook.com/10.35.11.29:0.logRoller] regionserver.LogRoller(176): LogRoller exiting. 2016-01-11 22:48:30,310 INFO [RS:1;dev1762:33025] regionserver.CompactSplitThread(403): Waiting for Split Thread to finish... 2016-01-11 22:48:30,310 INFO [RS:1;dev1762:33025] regionserver.CompactSplitThread(403): Waiting for Merge Thread to finish... 2016-01-11 22:48:30,310 INFO [RS:1;dev1762:33025] regionserver.CompactSplitThread(403): Waiting for Large Compaction Thread to finish... 2016-01-11 22:48:30,310 INFO [RS:1;dev1762:33025] regionserver.CompactSplitThread(403): Waiting for Small Compaction Thread to finish... 2016-01-11 22:48:30,310 INFO [regionserver/dev1762.prn2.facebook.com/10.35.11.29:0.leaseChecker] regionserver.Leases(146): regionserver/dev1762.prn2.facebook.com/10.35.11.29:0.leaseChecker closing leases 2016-01-11 22:48:30,310 INFO [regionserver/dev1762.prn2.facebook.com/10.35.11.29:0.leaseChecker] regionserver.Leases(146): regionserver/dev1762.prn2.facebook.com/10.35.11.29:0.leaseChecker closing leases 2016-01-11 22:48:30,311 INFO [regionserver/dev1762.prn2.facebook.com/10.35.11.29:0.leaseChecker] regionserver.Leases(149): regionserver/dev1762.prn2.facebook.com/10.35.11.29:0.leaseChecker closed leases 2016-01-11 22:48:30,311 INFO [regionserver/dev1762.prn2.facebook.com/10.35.11.29:0.leaseChecker] regionserver.Leases(149): regionserver/dev1762.prn2.facebook.com/10.35.11.29:0.leaseChecker closed leases 2016-01-11 22:48:30,317 INFO [dev1762.prn2.facebook.com,61792,1452581304511_ChoreService_1] hbase.ScheduledChore(179): Chore: dev1762.prn2.facebook.com,61792,1452581304511-MemstoreFlusherChore was stopped 2016-01-11 22:48:30,325 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(554): regionserver:33025-0x1523497b4040002, quorum=localhost:55120, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/replication/rs/dev1762.prn2.facebook.com,33025,1452581304563 2016-01-11 22:48:30,326 INFO [RS:1;dev1762:33025] ipc.RpcServer(2318): Stopping server on 33025 2016-01-11 22:48:30,326 INFO [RpcServer.listener,port=33025] ipc.RpcServer$Listener(815): RpcServer.listener,port=33025: stopping 2016-01-11 22:48:30,327 INFO [RpcServer.responder] ipc.RpcServer$Responder(1057): RpcServer.responder: stopped 2016-01-11 22:48:30,327 INFO [RpcServer.responder] ipc.RpcServer$Responder(960): RpcServer.responder: stopping 2016-01-11 22:48:30,333 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(554): master:49897-0x1523497b4040000, quorum=localhost:55120, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/dev1762.prn2.facebook.com,33025,1452581304563 2016-01-11 22:48:30,333 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(554): regionserver:61792-0x1523497b4040001, quorum=localhost:55120, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/dev1762.prn2.facebook.com,33025,1452581304563 2016-01-11 22:48:30,333 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(554): regionserver:33025-0x1523497b4040002, quorum=localhost:55120, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/dev1762.prn2.facebook.com,33025,1452581304563 2016-01-11 22:48:30,333 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(554): regionserver:61792-0x1523497b4040001, quorum=localhost:55120, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2016-01-11 22:48:30,333 INFO [main-EventThread] zookeeper.RegionServerTracker(118): RegionServer ephemeral node deleted, processing expiration [dev1762.prn2.facebook.com,33025,1452581304563] 2016-01-11 22:48:30,334 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(554): regionserver:33025-0x1523497b4040002, quorum=localhost:55120, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2016-01-11 22:48:30,334 INFO [main-EventThread] master.ServerManager(612): Cluster shutdown set; dev1762.prn2.facebook.com,33025,1452581304563 expired; onlineServers=1 2016-01-11 22:48:30,335 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(554): master:49897-0x1523497b4040000, quorum=localhost:55120, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2016-01-11 22:48:30,339 DEBUG [RS_CLOSE_REGION-dev1762:61792-1] regionserver.HRegionFileSystem(381): Committing store file hdfs://localhost:59295/user/elliott/test-data/4f0ddc3b-9bf4-4ec6-b2ab-29e91003a033/data/hbase/namespace/8fdaad26859458841be03061163dcfe7/.tmp/00c3373677094307a4be61e0469ea596 as hdfs://localhost:59295/user/elliott/test-data/4f0ddc3b-9bf4-4ec6-b2ab-29e91003a033/data/hbase/namespace/8fdaad26859458841be03061163dcfe7/info/00c3373677094307a4be61e0469ea596 2016-01-11 22:48:30,342 INFO [RS:1;dev1762:33025] regionserver.HRegionServer(1104): stopping server dev1762.prn2.facebook.com,33025,1452581304563; zookeeper connection closed. 2016-01-11 22:48:30,342 INFO [RS:1;dev1762:33025] regionserver.HRegionServer(1107): RS:1;dev1762:33025 exiting 2016-01-11 22:48:30,342 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@94e94ee] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(191): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@94e94ee 2016-01-11 22:48:30,348 INFO [RS_CLOSE_REGION-dev1762:61792-1] regionserver.HStore(968): Added hdfs://localhost:59295/user/elliott/test-data/4f0ddc3b-9bf4-4ec6-b2ab-29e91003a033/data/hbase/namespace/8fdaad26859458841be03061163dcfe7/info/00c3373677094307a4be61e0469ea596, entries=3, sequenceid=8, filesize=4.9 K 2016-01-11 22:48:30,350 INFO [RS_CLOSE_REGION-dev1762:61792-1] regionserver.HRegion(2454): Finished memstore flush of ~544 B/544, currentsize=0 B/0 for region hbase:namespace,,1452581306381.8fdaad26859458841be03061163dcfe7. in 122ms, sequenceid=8, compaction requested=false 2016-01-11 22:48:30,353 INFO [StoreCloserThread-hbase:namespace,,1452581306381.8fdaad26859458841be03061163dcfe7.-1] regionserver.HStore(875): Closed info 2016-01-11 22:48:30,358 DEBUG [RS_CLOSE_REGION-dev1762:61792-1] wal.WALSplitter(729): Wrote region seqId=hdfs://localhost:59295/user/elliott/test-data/4f0ddc3b-9bf4-4ec6-b2ab-29e91003a033/data/hbase/namespace/8fdaad26859458841be03061163dcfe7/recovered.edits/11.seqid to file, newSeqId=11, maxSeqId=2 2016-01-11 22:48:30,359 INFO [RS_CLOSE_REGION-dev1762:61792-1] regionserver.HRegion(1519): Closed hbase:namespace,,1452581306381.8fdaad26859458841be03061163dcfe7. 2016-01-11 22:48:30,359 DEBUG [RS_CLOSE_REGION-dev1762:61792-1] handler.CloseRegionHandler(164): Closed hbase:namespace,,1452581306381.8fdaad26859458841be03061163dcfe7. 2016-01-11 22:48:30,439 ERROR [RS:0;dev1762:61792] regionserver.HRegionServer(2822): Received CLOSE for a region which is not online, and we're not opening. 2016-01-11 22:48:30,439 WARN [RS:0;dev1762:61792] regionserver.HRegionServer(2748): Failed to close TestSplitWalDataLoss:dataloss,,1452581307705.bf63661cd1760fafa0bdfe4be2fb3776. - ignoring and continuing org.apache.hadoop.hbase.NotServingRegionException: The region bf63661cd1760fafa0bdfe4be2fb3776 is not online, and is not opening. at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegion(HRegionServer.java:2825) at org.apache.hadoop.hbase.regionserver.HRegionServer.closeRegionIgnoreErrors(HRegionServer.java:2743) at org.apache.hadoop.hbase.regionserver.HRegionServer.waitOnAllRegionsToClose(HRegionServer.java:1278) at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:1059) at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.runRegionServer(MiniHBaseCluster.java:156) at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.access$000(MiniHBaseCluster.java:108) at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer$1.run(MiniHBaseCluster.java:140) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:360) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1594) at org.apache.hadoop.hbase.security.User$SecureHadoopUser.runAs(User.java:334) at org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.run(MiniHBaseCluster.java:138) at java.lang.Thread.run(Thread.java:745) 2016-01-11 22:48:30,439 INFO [RS:0;dev1762:61792] regionserver.HRegionServer(1284): We were exiting though online regions are not empty, because some regions failed closing 2016-01-11 22:48:30,439 INFO [RS:0;dev1762:61792] regionserver.HRegionServer(1060): stopping server dev1762.prn2.facebook.com,61792,1452581304511; all regions closed. 2016-01-11 22:48:30,440 DEBUG [RS:0;dev1762:61792] wal.FSHLog(1063): Closing WAL writer in /user/elliott/test-data/4f0ddc3b-9bf4-4ec6-b2ab-29e91003a033/WALs/dev1762.prn2.facebook.com,61792,1452581304511 2016-01-11 22:48:30,443 INFO [IPC Server handler 8 on 59295] blockmanagement.BlockManager(2383): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:38785 is added to blk_1073741831_1007{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-2f4161bb-a789-4c3e-bb22-3dbf1babdf25:NORMAL|RBW], ReplicaUnderConstruction[[DISK]DS-940d5c3a-1db1-4402-9e4c-a19c2cc6af02:NORMAL|RBW]]} size 83 2016-01-11 22:48:30,443 INFO [IPC Server handler 7 on 59295] blockmanagement.BlockManager(2383): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:46850 is added to blk_1073741831_1007 size 1498 2016-01-11 22:48:30,446 DEBUG [RS:0;dev1762:61792] wal.FSHLog(1021): Moved 1 WAL file(s) to /user/elliott/test-data/4f0ddc3b-9bf4-4ec6-b2ab-29e91003a033/oldWALs 2016-01-11 22:48:30,446 INFO [RS:0;dev1762:61792] wal.FSHLog(1024): Closed WAL: FSHLog dev1762.prn2.facebook.com%2C61792%2C1452581304511..meta:.meta(num 1452581306140) 2016-01-11 22:48:30,446 DEBUG [RS:0;dev1762:61792] wal.FSHLog(1063): Closing WAL writer in /user/elliott/test-data/4f0ddc3b-9bf4-4ec6-b2ab-29e91003a033/WALs/dev1762.prn2.facebook.com,61792,1452581304511 2016-01-11 22:48:30,450 INFO [IPC Server handler 2 on 59295] blockmanagement.BlockManager(2383): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:46850 is added to blk_1073741829_1005{blockUCState=UNDER_CONSTRUCTION, primaryNodeIndex=-1, replicas=[ReplicaUnderConstruction[[DISK]DS-2f4161bb-a789-4c3e-bb22-3dbf1babdf25:NORMAL|RBW], ReplicaUnderConstruction[[DISK]DS-940d5c3a-1db1-4402-9e4c-a19c2cc6af02:NORMAL|RBW]]} size 83 2016-01-11 22:48:30,450 INFO [IPC Server handler 5 on 59295] blockmanagement.BlockManager(2383): BLOCK* addStoredBlock: blockMap updated: 127.0.0.1:38785 is added to blk_1073741829_1005 size 2201 2016-01-11 22:48:30,453 DEBUG [RS:0;dev1762:61792] wal.FSHLog(1021): Moved 1 WAL file(s) to /user/elliott/test-data/4f0ddc3b-9bf4-4ec6-b2ab-29e91003a033/oldWALs 2016-01-11 22:48:30,453 INFO [RS:0;dev1762:61792] wal.FSHLog(1024): Closed WAL: FSHLog dev1762.prn2.facebook.com%2C61792%2C1452581304511.default:(num 1452581305237) 2016-01-11 22:48:30,453 DEBUG [RS:0;dev1762:61792] ipc.RpcClientImpl(1147): Stopping rpc client 2016-01-11 22:48:30,454 DEBUG [RpcServer.reader=1,bindAddress=dev1762.prn2.facebook.com,port=49897] ipc.RpcServer$Listener(911): RpcServer.listener,port=49897: DISCONNECTING client 10.35.11.29:45302 because read count=-1. Number of active connections: 1 2016-01-11 22:48:30,463 INFO [RS:0;dev1762:61792] regionserver.Leases(146): RS:0;dev1762:61792 closing leases 2016-01-11 22:48:30,463 INFO [RS:0;dev1762:61792] regionserver.Leases(149): RS:0;dev1762:61792 closed leases 2016-01-11 22:48:30,464 INFO [RS:0;dev1762:61792] hbase.ChoreService(323): Chore service for: dev1762.prn2.facebook.com,61792,1452581304511 had [[ScheduledChore: Name: MovedRegionsCleaner for region dev1762.prn2.facebook.com,61792,1452581304511 Period: 120000 Unit: MILLISECONDS]] on shutdown 2016-01-11 22:48:30,775 INFO [dev1762.prn2.facebook.com,49897,1452581304150_splitLogManager__ChoreService_1] hbase.ScheduledChore(179): Chore: SplitLogManager Timeout Monitor was stopped 2016-01-11 22:48:31,135 INFO [RS_OPEN_META-dev1762:61792-0-MetaLogRoller] regionserver.LogRoller(176): LogRoller exiting. 2016-01-11 22:48:31,158 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(554): regionserver:61792-0x1523497b4040001, quorum=localhost:55120, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/replication/rs/dev1762.prn2.facebook.com,61792,1452581304511 2016-01-11 22:48:31,159 INFO [RS:0;dev1762:61792] ipc.RpcServer(2318): Stopping server on 61792 2016-01-11 22:48:31,159 INFO [RpcServer.listener,port=61792] ipc.RpcServer$Listener(815): RpcServer.listener,port=61792: stopping 2016-01-11 22:48:31,160 INFO [RpcServer.responder] ipc.RpcServer$Responder(1057): RpcServer.responder: stopped 2016-01-11 22:48:31,160 INFO [RpcServer.responder] ipc.RpcServer$Responder(960): RpcServer.responder: stopping 2016-01-11 22:48:31,193 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(554): regionserver:61792-0x1523497b4040001, quorum=localhost:55120, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/dev1762.prn2.facebook.com,61792,1452581304511 2016-01-11 22:48:31,193 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(554): master:49897-0x1523497b4040000, quorum=localhost:55120, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/dev1762.prn2.facebook.com,61792,1452581304511 2016-01-11 22:48:31,194 INFO [main-EventThread] zookeeper.RegionServerTracker(118): RegionServer ephemeral node deleted, processing expiration [dev1762.prn2.facebook.com,61792,1452581304511] 2016-01-11 22:48:31,194 INFO [main-EventThread] master.ServerManager(612): Cluster shutdown set; dev1762.prn2.facebook.com,61792,1452581304511 expired; onlineServers=0 2016-01-11 22:48:31,194 DEBUG [M:0;dev1762:49897] master.HMaster(1125): Stopping service threads 2016-01-11 22:48:31,208 INFO [RS:0;dev1762:61792] regionserver.HRegionServer(1104): stopping server dev1762.prn2.facebook.com,61792,1452581304511; zookeeper connection closed. 2016-01-11 22:48:31,209 INFO [RS:0;dev1762:61792] regionserver.HRegionServer(1107): RS:0;dev1762:61792 exiting 2016-01-11 22:48:31,209 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@6add7ab0] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(191): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@6add7ab0 2016-01-11 22:48:31,210 INFO [main] util.JVMClusterUtil(317): Shutdown of 1 master(s) and 2 regionserver(s) complete 2016-01-11 22:48:31,217 DEBUG [main-EventThread] zookeeper.ZooKeeperWatcher(554): master:49897-0x1523497b4040000, quorum=localhost:55120, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2016-01-11 22:48:31,217 INFO [M:0;dev1762:49897] client.ConnectionManager$HConnectionImplementation(1712): Closing zookeeper sessionid=0x1523497b4040006 2016-01-11 22:48:31,217 DEBUG [main-EventThread] zookeeper.ZKUtil(370): master:49897-0x1523497b4040000, quorum=localhost:55120, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2016-01-11 22:48:31,225 DEBUG [M:0;dev1762:49897] ipc.RpcClientImpl(1147): Stopping rpc client 2016-01-11 22:48:31,236 INFO [M:0;dev1762:49897] hbase.ChoreService(323): Chore service for: dev1762.prn2.facebook.com,49897,1452581304150_splitLogManager_ had [] on shutdown 2016-01-11 22:48:31,236 INFO [M:0;dev1762:49897] flush.MasterFlushTableProcedureManager(79): stop: server shutting down. 2016-01-11 22:48:31,237 INFO [M:0;dev1762:49897] ipc.RpcServer(2318): Stopping server on 49897 2016-01-11 22:48:31,237 INFO [RpcServer.listener,port=49897] ipc.RpcServer$Listener(815): RpcServer.listener,port=49897: stopping 2016-01-11 22:48:31,237 INFO [RpcServer.responder] ipc.RpcServer$Responder(1057): RpcServer.responder: stopped 2016-01-11 22:48:31,237 INFO [RpcServer.responder] ipc.RpcServer$Responder(960): RpcServer.responder: stopping 2016-01-11 22:48:31,241 DEBUG [M:0;dev1762:49897] zookeeper.RecoverableZooKeeper(188): Node /hbase/rs/dev1762.prn2.facebook.com,49897,1452581304150 already deleted, retry=false 2016-01-11 22:48:31,250 INFO [M:0;dev1762:49897] regionserver.HRegionServer(1104): stopping server dev1762.prn2.facebook.com,49897,1452581304150; zookeeper connection closed. 2016-01-11 22:48:31,250 INFO [M:0;dev1762:49897] regionserver.HRegionServer(1107): M:0;dev1762:49897 exiting 2016-01-11 22:48:31,252 INFO [main] zookeeper.MiniZooKeeperCluster(316): Shutdown MiniZK cluster with all ZK servers 2016-01-11 22:48:31,252 WARN [main] datanode.DirectoryScanner(375): DirectoryScanner: shutdown has been called 2016-01-11 22:48:31,256 INFO [main] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2016-01-11 22:48:31,351 DEBUG [dev1762:49897.activeMasterManager-EventThread] zookeeper.ZooKeeperWatcher(554): replicationLogCleaner-0x1523497b4040007, quorum=localhost:55120, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Disconnected, path=null 2016-01-11 22:48:31,352 DEBUG [dev1762:49897.activeMasterManager-EventThread] zookeeper.ZooKeeperWatcher(643): replicationLogCleaner-0x1523497b4040007, quorum=localhost:55120, baseZNode=/hbase Received Disconnected from ZooKeeper, ignoring 2016-01-11 22:48:31,359 WARN [DataNode: [[[DISK]file:/data/users/elliott/hbase/hbase-server/target/test-data/106c9698-d5e9-426f-9886-d717fa502d54/dfscluster_3e875bf4-f4b9-453c-966d-153d9ade8758/dfs/data/data3/, [DISK]file:/data/users/elliott/hbase/hbase-server/target/test-data/106c9698-d5e9-426f-9886-d717fa502d54/dfscluster_3e875bf4-f4b9-453c-966d-153d9ade8758/dfs/data/data4/]] heartbeating to localhost/127.0.0.1:59295] datanode.BPServiceActor(722): BPOfferService for Block pool BP-13867801-10.35.11.29-1452581300662 (Datanode Uuid 13b3e9de-5a2b-47a5-b897-0039ad33a71c) service to localhost/127.0.0.1:59295 interrupted 2016-01-11 22:48:31,360 WARN [DataNode: [[[DISK]file:/data/users/elliott/hbase/hbase-server/target/test-data/106c9698-d5e9-426f-9886-d717fa502d54/dfscluster_3e875bf4-f4b9-453c-966d-153d9ade8758/dfs/data/data3/, [DISK]file:/data/users/elliott/hbase/hbase-server/target/test-data/106c9698-d5e9-426f-9886-d717fa502d54/dfscluster_3e875bf4-f4b9-453c-966d-153d9ade8758/dfs/data/data4/]] heartbeating to localhost/127.0.0.1:59295] datanode.BPServiceActor(845): Ending block pool service for: Block pool BP-13867801-10.35.11.29-1452581300662 (Datanode Uuid 13b3e9de-5a2b-47a5-b897-0039ad33a71c) service to localhost/127.0.0.1:59295 2016-01-11 22:48:31,361 WARN [main] datanode.DirectoryScanner(375): DirectoryScanner: shutdown has been called 2016-01-11 22:48:31,362 INFO [main] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2016-01-11 22:48:31,465 WARN [DataNode: [[[DISK]file:/data/users/elliott/hbase/hbase-server/target/test-data/106c9698-d5e9-426f-9886-d717fa502d54/dfscluster_3e875bf4-f4b9-453c-966d-153d9ade8758/dfs/data/data1/, [DISK]file:/data/users/elliott/hbase/hbase-server/target/test-data/106c9698-d5e9-426f-9886-d717fa502d54/dfscluster_3e875bf4-f4b9-453c-966d-153d9ade8758/dfs/data/data2/]] heartbeating to localhost/127.0.0.1:59295] datanode.BPServiceActor(722): BPOfferService for Block pool BP-13867801-10.35.11.29-1452581300662 (Datanode Uuid 83780db6-1977-4d15-a635-31f452ae5cf3) service to localhost/127.0.0.1:59295 interrupted 2016-01-11 22:48:31,466 WARN [DataNode: [[[DISK]file:/data/users/elliott/hbase/hbase-server/target/test-data/106c9698-d5e9-426f-9886-d717fa502d54/dfscluster_3e875bf4-f4b9-453c-966d-153d9ade8758/dfs/data/data1/, [DISK]file:/data/users/elliott/hbase/hbase-server/target/test-data/106c9698-d5e9-426f-9886-d717fa502d54/dfscluster_3e875bf4-f4b9-453c-966d-153d9ade8758/dfs/data/data2/]] heartbeating to localhost/127.0.0.1:59295] datanode.BPServiceActor(845): Ending block pool service for: Block pool BP-13867801-10.35.11.29-1452581300662 (Datanode Uuid 83780db6-1977-4d15-a635-31f452ae5cf3) service to localhost/127.0.0.1:59295 2016-01-11 22:48:31,474 WARN [org.apache.hadoop.hdfs.server.blockmanagement.DecommissionManager$Monitor@3f053c80] blockmanagement.DecommissionManager$Monitor(78): Monitor interrupted: java.lang.InterruptedException: sleep interrupted 2016-01-11 22:48:31,487 INFO [main] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2016-01-11 22:48:31,606 INFO [main] hbase.HBaseTestingUtility(1103): Minicluster is down 2016-01-11 22:48:31,643 INFO [main] hbase.ResourceChecker(171): after: regionserver.TestSplitWalDataLoss#test Thread=24 (was 4) Potentially hanging thread: snapshot-hfile-cleaner-cache-refresher java.lang.Object.wait(Native Method) java.util.TimerThread.mainLoop(Timer.java:552) java.util.TimerThread.run(Timer.java:505) Potentially hanging thread: IPC Client (1393385147) connection to localhost/127.0.0.1:59295 from elliott java.lang.Object.wait(Native Method) org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:904) org.apache.hadoop.ipc.Client$Connection.run(Client.java:949) Potentially hanging thread: process reaper sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: Timer for 'NameNode' metrics system java.lang.Object.wait(Native Method) java.util.TimerThread.mainLoop(Timer.java:552) java.util.TimerThread.run(Timer.java:505) Potentially hanging thread: LeaseRenewer:elliott@localhost:59295 java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.LeaseRenewer.run(LeaseRenewer.java:438) org.apache.hadoop.hdfs.LeaseRenewer.access$700(LeaseRenewer.java:71) org.apache.hadoop.hdfs.LeaseRenewer$1.run(LeaseRenewer.java:298) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: dev1762:49897.activeMasterManager-EventThread sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494) Potentially hanging thread: snapshot-log-cleaner-cache-refresher java.lang.Object.wait(Native Method) java.util.TimerThread.mainLoop(Timer.java:552) java.util.TimerThread.run(Timer.java:505) Potentially hanging thread: IPC Parameter Sending Thread #1 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: LeaseRenewer:elliott.hfs.0@localhost:59295 java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.LeaseRenewer.run(LeaseRenewer.java:438) org.apache.hadoop.hdfs.LeaseRenewer.access$700(LeaseRenewer.java:71) org.apache.hadoop.hdfs.LeaseRenewer$1.run(LeaseRenewer.java:298) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: HBase-Metrics2-1 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: IPC Parameter Sending Thread #0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: org.apache.hadoop.hdfs.PeerCache@4dc9417f java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.PeerCache.run(PeerCache.java:244) org.apache.hadoop.hdfs.PeerCache.access$000(PeerCache.java:41) org.apache.hadoop.hdfs.PeerCache$1.run(PeerCache.java:119) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: LeaseRenewer:elliott.hfs.1@localhost:59295 java.lang.Thread.sleep(Native Method) org.apache.hadoop.hdfs.LeaseRenewer.run(LeaseRenewer.java:438) org.apache.hadoop.hdfs.LeaseRenewer.access$700(LeaseRenewer.java:71) org.apache.hadoop.hdfs.LeaseRenewer$1.run(LeaseRenewer.java:298) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: IPC Client (1393385147) connection to localhost/127.0.0.1:59295 from elliott.hfs.0 java.lang.Object.wait(Native Method) org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:904) org.apache.hadoop.ipc.Client$Connection.run(Client.java:949) Potentially hanging thread: process reaper sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: IPC Client (1393385147) connection to localhost/127.0.0.1:59295 from elliott.hfs.1 java.lang.Object.wait(Native Method) org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:904) org.apache.hadoop.ipc.Client$Connection.run(Client.java:949) Potentially hanging thread: SessionTracker java.lang.Object.wait(Native Method) org.apache.zookeeper.server.SessionTrackerImpl.run(SessionTrackerImpl.java:146) Potentially hanging thread: IPC Client (1393385147) connection to localhost/127.0.0.1:59295 from elliott java.lang.Object.wait(Native Method) org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:904) org.apache.hadoop.ipc.Client$Connection.run(Client.java:949) Potentially hanging thread: region-location-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081) java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) java.lang.Thread.run(Thread.java:745) Potentially hanging thread: dev1762:49897.activeMasterManager-SendThread(localhost:55120) java.lang.Thread.sleep(Native Method) org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:994) - Thread LEAK? -, OpenFileDescriptor=252 (was 166) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=65535 (was 65535), SystemLoadAverage=286 (was 312), ProcessCount=562 (was 561) - ProcessCount LEAK? -, AvailableMemoryMB=8855 (was 9633), ConnectionCount=0 (was 0) 2016-01-11 22:48:31,647 INFO [Thread-2] regionserver.ShutdownHook$ShutdownHookThread(111): Shutdown hook starting; hbase.shutdown.hook=true; fsShutdownHook=org.apache.hadoop.fs.FileSystem$Cache$ClientFinalizer@657c8ad9 2016-01-11 22:48:31,647 INFO [Thread-2] regionserver.ShutdownHook$ShutdownHookThread(133): Shutdown hook finished. 2016-01-11 22:48:31,647 INFO [Thread-2] regionserver.ShutdownHook$ShutdownHookThread(111): Shutdown hook starting; hbase.shutdown.hook=true; fsShutdownHook=org.apache.hadoop.fs.FileSystem$Cache$ClientFinalizer@657c8ad9 2016-01-11 22:48:31,647 INFO [Thread-2] regionserver.ShutdownHook$ShutdownHookThread(120): Starting fs shutdown hook thread. 2016-01-11 22:48:31,647 INFO [Thread-2] regionserver.ShutdownHook$ShutdownHookThread(133): Shutdown hook finished.